Using GenAI in HR Compliance Requires Collaboration, Validation

What happened when Mitratech asked ChatGPT a series of common HR questions, and pitted the answers against the software development company’s human experts, who individually boast an average of 18 years of experience?

Susan Anderson, SHRM-SCP, chief services officer at Mitratech in Bee Cave, Texas, shared her team's findings during a June 24 concurrent session at SHRM24 in Chicago.

The results were mixed, according to Anderson, who has led the development of proprietary HR compliance custom GPTs for research, writing, and delivery of services and leads the company's team of HR experts. 

While surveys and polls a year ago were indicating HR’s skepticism about AI use in the workplace, that has changed, she said, referring to the 2024 Work Trend Index Annual Report from Microsoft and LinkedIn. The report is based on a survey of 31,000 workers worldwide conducted in February and March.

“The data is in: 2024 is the year AI at work gets real,” the report said. “Use of generative AI [GenAI] has nearly doubled in the last six months, with 75% of global knowledge workers using it. And employees, struggling under the pace and volume of work, are bringing their own AI to work.”

Among workers using AI, 52% are reluctant to admit using it for their most important tasks, according to the report. In addition, “over half of people using it are not only afraid to tell you [they’re using it], they’re afraid they’re going to lose their job or [their boss is] going to suddenly double their workload,” Anderson said.

However, she predicted that “AI won’t displace experts, but it will displace those not using AI.” 

What They Learned

Anderson and her team set out to determine what AI could do well regarding HR compliance and where it was lacking. They scored GenAI responses to their questions based on accuracy, context relevancy, consistency of answers, brevity, bias level, and practicality.

“In most cases, the answers weren’t comprehensive,” she said. “Sometimes they were accurate but lacked nuance humans can provide,” which she attributed in part to how the team crafted the prompt questions.

Lessons learned: Prioritize your team’s training, set guidelines, be watchful of ethical boundaries, and focus on improving prompts. Asking the wrong question is risky, Anderson said, and she warned attendees at the packed session to “beware of answers that sound right.”

“When asking questions, validate elsewhere. You have to have an expert for context and know if the output you’re getting is even valid.”

Be bold and experiment with GenAI, but set guidelines for how you use the tool, your data, and what you do with the output, Anderson said.

In another experiment, Anderson’s team applied ChatGPT to the advisory team’s written responses to clients. What was the speed and quality of the responses? What was the impact on the employee’s experience using these tools? What was the impact on the customer experience?

Among the key findings:

• GenAI is useful for best practice and strategy-based questions.
• It aids in answering simple yes/no questions and expanding on reasoning behind answers.
• It helps decipher unclear client questions for better initial response formulation. ChatGPT responded well to questions about background checks, exempt/nonexempt classifications, termination decision-making, drug and alcohol issues, workers’ compensation, compensation practices, and payroll and tax issues.
• GenAI enhanced the efficiency and quality of responses to clients.

It’s important to empower your workforce to experiment and grow with AI, Anderson emphasized. This is done by:

• Establishing a solid foundation and building the knowledge and skills to experiment with this tool and overcome resistance to using it.
  Provide dedicated resources and offer continuous learning, including training for managers, having AI champions for different departments, and schooling workers on data privacy and establishing prompting techniques.
• Fostering a culture of experimentation. Do this by establishing “safe to fail” environments.

“We wanted failures to be public and celebrated, like wins,” she said. Employees were encouraged to show off their work, as well as talk about how they were following the guidelines the company established for using AI.

“Collaborative intelligence is the future,” she told attendees as the session concluded.