Reporting on Training

Tamar Elkeles is a numbers-cruncher. She knows exactly who her customers are and how they are consuming her product. She knows how much each customer interaction costs and what it brings to the bottom line. And since she knows that her success is measured by her results, each year she compiles an annual report that is shared with stakeholders.

Elkeles isn’t an accountant or a CFO—she’s vice president of learning and organizational development at QUALCOMM Inc., a San Diego-based wireless communications company. Her customers are some 9,300 QUALCOMM employees around the world; her product, the more than 500 different programs she and her staff develop and deliver each year.

Elkeles sees the time spent on the annual report as an investment. “It’s important to demonstrate the value of learning,” she says.

Experts say annual reports are a growing trend in learning organizations. “Training is now being held to the same standard as other [business] segments,” says Paul Sanchez, global director of employee research at Mercer Human Resource Consulting in New York. Learning professionals at all levels are now often asked to demonstrate training’s impact on business goals and to conduct return on investment (ROI) analysis. A formal annual report on training activity and results takes the idea one step further, consolidating individual analyses into one comprehensive report.

Plus, proactively compiling a report can help learning professionals make a positive impression on their organization’s senior management. “Do it before someone asks you for it,” says Leslie Joyce, chief learning officer at Atlanta-based home improvement giant The Home Depot. “Then they know you are really a business partner.”

Calculate and Communicate

Even the smallest training departments do some sort of post-training evaluation, and maybe even an occasional ROI analysis. But many departments lack a systematic approach to analysis and a regular means of communicating the results to senior management. In today’s competitive environment with each business function vying for its share of the budget, that can be a mistake.

“Every company today is looking for return on how it invests resources,” says Sanchez. “Training as a staff function absorbs a huge percentage of the revenue dollar. Companies want to know if the investment is being appropriately applied, and if there are effective, measurable results.”

A training annual report can answer both needs. First, it forces training managers to develop a systematic analysis approach, so that they can clearly demonstrate their results. Second, it provides a vehicle for communicating those results to senior management. “An annual report can absolutely help [training managers] with gaining a seat at the table,” says Theresa Seagraves, a training value and ROI expert and author of Quick! Show Me Your value (ASTD Press, 2004). “It provides the basis for an excellent conversation.”

Training professionals know that their budgets are often the first to suffer when business takes a downturn. Consistently reporting data on training’s positive effect on business goals can help break that pattern and put training managers in a stronger position to negotiate and defend their budgets, says Seagraves.

Learning professionals in the trenches agree. “If you don’t report on what you’re doing, people will say, ‘Of course [training is] valuable, but I don’t know if I can afford it this quarter,’ ” says Ted Hoff, chief learning officer at IBM in White Plains, N.Y. “When you have this kind of disciplined reporting, the same discipline you bring to a capital investment, people will follow up on the commitment” to training initiatives.

Discipline Boosts Departmental Value

But compiling a training annual report isn’t just an exercise in justification. Learning professionals say that sharing the report with training staff can reap departmental benefits as well. “The value comes from the discipline of running learning like a business,” says David Vance, president of Caterpillar University, the learning organization of Peoria, Ill.-based heavy equipment and engine manufacturer Caterpillar.

For instance, at The Home Depot, new cashier training used to be a blend of classroom training and two hours of hands-on practice in a simulated work environment. But the annual analysis showed that cashier effectiveness scores were mainly influenced by the classroom training portion, while the lab portion added little value. As a result, that portion of the training was eliminated. “We saved dollars by returning individuals to the floor two hours earlier,” says Joyce.

An annual review of the training department’s impact also can go a long way toward boosting morale and motivating employees who design and produce learning initiatives. “When people in the learning community saw [our results], it really did a great deal to motivate them,” says Joyce. “Suddenly your role is very concrete: Something you wrote changed the bottom line.”

Starting from Scratch

Some learning organizations already do analysis and reporting regularly for their own use; in these cases, it’s fairly easy to take the extra step of summarizing the data annually into a report for senior management. But training departments with no analysis and reporting foundation may be intimidated by the prospect.

“People are always afraid of the time investment,” says Seagraves. “It can be time-consuming—it’s hard for people to commit to that first time. But after you’ve done [the annual report] once, it becomes very easy to update.”

There are no hard-and-fast rules on the content of a training annual report, but as a starting point, Sanchez recommends following a basic outline:

• Review the company’s broader HR mission and the major issues facing the company that have been addressed through training—for example, reducing accidents in the workplace through safety training, or focusing on loss prevention through specialized training.
• Describe the training function’s goals and objectives for the past year. Specify the indicators used to measure the success or failure of each goal and report the department’s performance using these indicators.
• Provide an overall, realistic assessment against the budget. Demonstrate commitment to savings wherever possible. At The Home Depot, Joyce’s annual review includes data about any projects that have reduced the cost of learning development. “If the days to development are reduced, we know the cost savings of getting [curriculum] to stores a week earlier,” says Joyce. “If we turn an instructor-led module into an e-learning module, we calculate out the cost savings and learning impact.”
• Offer conclusions drawn from the past year’s performance and give recommendations for improvement for the coming year.

Digging Out the Data

Collecting the necessary data can be the biggest challenge in compiling a training annual report. While Hoff says IBM’s data capture and analysis process is largely automated through its learning management system, Joyce says much of the capture and analysis conducted at The Home Depot is done by hand. She says the company calculates such things as training programs’ ROI, impact on sales, and effect on turnover, losses and accidents.

At Caterpillar University, Vance and his team do some of the analysis manually as well—and he wouldn’t have it any other way. Vance says that much of the value of the annual report exercise comes in giving training managers a better grasp of the data surrounding their work. Having the analysis automated “would defeat some of the benefit—it’s like skipping your homework and someone gives you the answers,” Vance says.

Departments without readily available data can ramp up by embedding measurement indicators at all levels of learning. For each program, determine how you will evaluate its effectiveness. Then build a system to track those indicators over time—whether the data is part of a learning management system or is tracked using a simple Excel spreadsheet. For new programs, identify the measurements that will be used to quantify results, says Joyce.

If this all seems overwhelming, Vance recommends starting small. “You don’t have to do it for every program,” he says. Start with the top 10 training programs. Measure the programs based on Donald Kirkpatrick’s four levels of evaluation. (For more explanation on this analysis, see “Evaluating Evaluations” in the June 2002 issue of HR Magazine.) “We’re not talking a lot of money, or a dedicated person,” to measure the programs, says Vance. He says that even companies that don’t have a computer-based learning management system, or that track training data using an Excel worksheet, can benefit from “just starting the process.”

Don’t be discouraged by inexact data. “People want it to be perfect,” says Joyce. “The fact that you’re looking at it and thinking about it puts you eons ahead of people who aren’t even asking the questions yet.” Vance agrees: “It doesn’t have to be super accurate; you’re just trying to get in the ballpark here.”

Know Your Audience

When identifying what areas to highlight in a training annual report, be aware of your audience. “See what’s important to your chief executive,” says Seagraves. If the CEO is focused on the effect of workplace accidents on manufacturing output, highlight how safety training has reduced the number of accidents in the past year, and how productivity has increased.

If the executive spotlight is on increasing sales, focus on how targeted product knowledge training or sales training has boosted results. Seek out testimonial statements from managers who have seen training results firsthand. “It is always much better if someone else is communicating your value for you,” says Seagraves.

While it’s important to demonstrate the learning organization’s value and impact, don’t get too bogged down in data. “A lot of people won’t read it all,” says Seagraves. She recommends providing a one-page executive summary and succinct, one-line section headings to get the main points across in a quick scan. Then provide the heavy-duty data in backup sections.

“What [CEOs] want to know is, are you focusing [the training] in the right places? Can you take all the programs you’re working on and tell me how [reaching business goals] is going to happen?” Seagraves says.

IBM’s annual training report for executive management summarizes the activities and results that more than 1,200 learning professionals delivered to some 350,000 employees around the world. In the report, Hoff says, “we give aggregated results across all learning on certain key questions.” Examples of those questions include:

• How much did we invest in training?
• How much time did it take to conduct the training?
• How much value was realized from each training dollar spent?
• Across all courses, what did participants think about the programs?

The leaders of each of IBM’s six business units and each geographic region receive more-detailed reports that focus on training activities and results in their unit or region. “We give as much depth as is needed for the level of executive,” says Hoff.

Talking ’Bout an Evolution

Despite the label, writing an annual report is not really an annual event—it’s an ongoing process throughout the year. Most organizations track training data continually, with periodic review and reporting monthly, quarterly or semi-annually. These periodic reports come in handy in the third quarter—the time when most executives begin working on corporate annual reports.

Deciding to compile a training annual report is also a philosophical change for a learning organization—and, like any change, it takes time. “It’s an evolutionary process,” says Vance. “There aren’t any silver bullets here.”

But experts and experienced learning professionals agree the time and effort are well worth it. “If you want to be in the game, you need to have this groundwork laid,” says Seagraves. “Doing an annual report is an excellent way to lay the groundwork and get ahead of the pack.”

Jennifer Taylor Arnold is a freelance writer in Baltimore.