LinkedIn’s ‘Model Drift’ Experience Provides Lesson for HR

By Nicole Lewis June 22, 2021
LIKE SAVE
woman studying data

​When a LinkedIn team developed an artificial intelligence algorithm to boost the company's forecasting capabilities, its members couldn't have known that the coronavirus pandemic would upend everything and create "model drift"—a discrepancy between old and new data that causes prediction errors and forecasting flaws.

LinkedIn's experience is an example of what analysts expect to see as HR executives increasingly apply AI tools to HR data that can quickly become outdated, shifting forecasts and disrupting business plans.

LinkedIn's COVID-19 Data Disruption

Eighteen months ago, before the coronavirus disrupted the global workforce, executives at LinkedIn began using machine learning to automate forecasting for its sales and marketing teams to help them identify likely buyers of company products.

Combing through the data of more than 756 million global members is an arduous task, made more difficult when manual processes are involved, said Jilei Yang, senior software engineer at LinkedIn.

"Prior to this tool, the sales and marketing teams would target consumers based on their own experience," she said. "They would look through tens of thousands of accounts every day and manually pick out some accounts they thought might be promising. That wasn't a very efficient or scalable process."  

To better identify likely buyers of LinkedIn's talent, marketing, sales and subscription products, which are core revenue-generating offerings, LinkedIn developed a prediction model that was trained on two years of customer data.

The system monitors metrics such as how many people sign up on LinkedIn each day, how many use LinkedIn every hour, day, week or month and what products customers buy. Particular attributes of each company are also documented, such as whether a company is a small, medium or large business, or what products they may already be using.

As LinkedIn launched new products, executives hoped the system would give the sales and marketing team better predictions on which customers would provide the best opportunity to turn sales pitches into purchasing deals. 

Their hopes were dashed, however, when the COVID-19 pandemic hit and customers' spending behavior changed. Suddenly, forecasts became irrelevant. The two-year-old data was obsolete as customers halted operations, stopped hiring staff and were less likely to buy LinkedIn products.

 LinkedIn was facing model drift, which occurs when new data that influences the interpretation of AI algorithms doesn't resemble the data the algorithm was trained on.

"Before COVID-19, our labels would show us that certain LinkedIn customers may be willing to buy the LinkedIn recruiter product," Yang said. "However, during the pandemic their sentiments changed, and those customers were less willing to buy the product."

LinkedIn's team had to retrain the algorithm with fresh data based on interviews with customers. They spent five months collecting information and updating the systems to reflect more recent customer purchasing patterns. Working with new data, the sales and marketing teams could prioritize customers who were more likely to buy products during the pandemic.

As data grows exponentially and AI continues to be used to gain insights into employee productivity, the supply and demand of workers, skills shortages, telecommuting patterns, and compensation rates, analysts warn of growing model drift disruptions.

In a recent report titled Workforce and Learning Trends 2021, researchers at nonprofit trade association CompTIA state, "IT professionals will need to learn to spot and correct for model drift."

To help employers, companies like Amazon, IBM and StreamSets, Inc., offer tools to manage model drift, but HR managers can't depend on tools alone, said Ritu Jyoti, program vice president of worldwide artificial intelligence and automation research for IDC, a market intelligence and research firm in Needham, Mass.

Poor forecasting harms a company's reputation, brand and income, Jyoti observed. To avoid this, HR executives should involve heads of business units such as the chief information officer, the chief transformation officer and other department leaders when embarking on an AI forecasting project.

"My advice to HR managers implementing AI forecasting tools is to think about whether you have the right skills and talent and whether you have accountability and corporate governance mechanisms in place," Jyoti said.

Greykite

To help employers prepare for model drift, LinkedIn has launched an open-source Python library for companies that are looking for AI-driven forecasting tools. Dubbed Greykite, the platform is designed to help users forecast resource planning, performance management, optimization and other internal business processes.

Greykite can show employers, with the help of their engineering teams, changes in trends. The tool's built-in prediction models are fast and adaptive, executives say.

"It depends on the model or system you're using for training, but our whole model training and evaluation pipeline takes only a few seconds to train a model," said Reza Hosseini, staff applied software engineer at LinkedIn.

In human resource planning, where employers want to be one step ahead of their competitors, one area in which Greykite can make a difference is talent acquisition.

"Greykite can be used to track pools of talent at certain periods in certain areas of the country," Hosseini said. "However, they'll have to keep in mind that an unknown event can change the number of available candidates. The numbers can increase, and numbers can drop off."

Greykite makes predictions, but only with the data available, so it can't predict how big deviations like a global pandemic will change things until it sees the changes happening in the data.

Nicole Lewis is a freelance journalist based in Miami.

LIKE SAVE

SHRM HR JOBS

Hire the best HR talent or advance your own career.

Member Benefit: Ask-An-Advisor Service

SHRM's HR Knowledge Advisors offer guidance and resources to assist members with their HR inquiries.

SHRM's HR Knowledge Advisors offer guidance and resources to assist members with their HR inquiries.

REACH OUT NOW

SPONSOR OFFERS

HR Daily Newsletter

News, trends and analysis, as well as breaking news alerts, to help HR professionals do their jobs better each business day.