Evaluating AI-Based Assessments in Personnel Selection

​Talent management tools and technologies using artificial intelligence (AI) have proliferated in recent years, sold to organizations with the promise of improving strategies for recruitment, selection and development. 

There are concerns, though, about the inadequacies and potential biases of AI-based assessments. The Society for Industrial and Organizational Psychology (SIOP) offers guidance in a brief video, "AI and Selection," to help employers make realistic evaluations of these tools.  

In the presentation, Fred Oswald, professor of industrial organizational (I-O) psychology at Rice University, describes four main considerations surrounding the use of AI-based assessment tools.  

1. AI-based assessments should meet professional testing standards that reflect the consensus of professional opinion on fair and accurate assessments. Standards for the validation and use of personnel selection procedures and for educational and psychological testing can provide employers with guidance on the validity, reliability and fairness of AI-based tools and are available free online.
2.  AI-based assessments must be job-relevant, measuring the knowledge, skills, abilities or other characteristics (KSAOs) necessary to perform the job. While not all job-relevant KSAOs need to be measured, those irrelevant to the job should not be measured. Good data is more important than "big data."
3.  AI-based assessments should minimize bias in the score differences among various demographic groups. Still, employers should demand more than lack of bias—after all, a coin flip is an unbiased selection method. If a tool is to be both fair and effective, it also must be job-relevant (see No. 2).
4. AI-based assessments are multifaceted. To minimize bias and error, employers should consider and evaluate several of these tools' components and aspects, which include:
   
   • Interactive technologies that job applicants will use with the tool, such as games, video interviews and the like.
   • How data will be extracted and transformed prior to being submitted to machine learning.
   • Actual machine-learning algorithms and data-mining techniques that will be applied to the data.
   • Scores that will emerge from the machine-learning analyses.
     
     

This video is part of the series "Spotlight on Science: Applying Research Outcomes for Better Workplaces," a repository of evidence-based HR resources SIOP produced in partnership with SHRM. These materials present data that infuse the science of I-O psychology and other HR disciplines into management practices for daily use.