By Jack J. Phillips and Patricia Pulliam Phillips
2005, 210 pages, Paperback with CD-ROM
SHRMStore Item: 61.16503
Order from SHRMStore or call (800) 444-5006
Preparing for ROI
Preparing the organization and the HR department for ROI measurement invites a few challenges. Therefore, reserve ROI analysis for certain HR programs in the organization. This chapter explores how to select programs for ROI analysis and describes the preparations necessary to begin each ROI study.
When to Use ROI
Every HR program should be evaluated in some way even if the evaluation involves only reaction/satisfaction data collected from a select group of stakeholders. Including evaluation as a part of every HR program is a fundamental requirement. Reaction/satisfaction data alone are sufficient for evaluating many HR programs. Nevertheless, when appropriate and feasible, you should collect additional data to conduct higher-level evaluations.
Decide upon the appropriate level of evaluation for each HR program when the program is initiated, realizing that the evaluation level may change throughout the life of the program. For example, you can evaluate a new-employee orientation program with reaction/satisfaction (level 1) and learning (level 2) data. The objective at this point is to ensure that employees learn what they need to know to be successful and have the appropriate impressions about the organization. Later, a follow-up evaluation may be implemented to determine if the new employees are using the information provided in the orientation and if they are progressing appropriately based on what they have learned during orientation. Such a follow-up evaluation is an application/implementation evaluation (level 3).
If the cost of the new employee orientation rises, some managers may question the value of the process and push the evaluation to higher levels. To carry out level 4 or 5 evaluations, it may be necessary to redesign the new-employee orientation program. You can then compare the redesigned version to the previous version to see if there is a difference. The difference may be evaluated with business impact data (level 4) or ROI (level 5) or both. During the life of a particular program or function, the desired level of evaluation may change.
Because of the resources required and the realistic barriers for implementing ROI, use the methodology only for certain programs. You should consider several criteria, outlined in the following sections, when selecting an evaluation level for HR programs.
Recommended Programs for Evaluation at Lower Levels
Reaction and satisfaction evaluation (level 1) can suffice as the only level of evaluation for short programs, such as briefings, policy introductions, or general information distributed to employees. If satisfaction with the program is critical, ongoing assessment of reaction may be appropriate. For example, a routine assessment of reaction to a telecommuting program is appropriate to determine the extent to which employees continue to perceive the program as fair, responsive, appropriate, and helpful.
Learning evaluation (level 2) is appropriate when it is essential for employees to acquire specific knowledge or skills presented in an HR program. Learning evaluations may be helpful for certain compliance initiatives in which employees must learn how and why the organization complies with a variety of regulations. These data are also critical for safety and health programs that safeguard employees’ welfare. Participants in safety programs must know correct procedures and practices. Learning evaluation is also helpful for certain policies that employees must know, such as diversity, value systems, and routine policies and procedures that are critical to the job and the organization.
Application and implementation evaluation (level 3) is necessary when it is important for employees to perform in a particular way on the job as a result of an HR program. For example, if employees are required to deliver a certain level of customer service based on job design, reward systems, or training and development, it may be helpful to monitor employees to be sure that they are delivering the proper service. Spot audits or observations of employees interacting with customers can often ensure they are applying appropriate behavior on the job. If safety and security are issues, it is absolutely essential that key programs be evaluated at level 3 to make certain that employees are doing what they are supposed to do or reacting properly under particular circumstances.
The tradeoff in deciding which level of evaluation is appropriate is not only a test of the ideal, but also a tradeoff with resources available and the amount of disruption allowed in the organization. Because most data collection at this level disrupts the organization in some way, the evaluation must be balanced with the time, effort, and resources that can be committed to the process. Most organizations fall short of the ideal evaluation and settle for a feasible approach within the existing constraints.
Recommended Programs for Impact and ROI Analysis
Programs taken to the levels of business impact and ROI analysis are special ones for which it is important to understand the contributions they make to the organization. Among the criteria you should consider when selecting programs to evaluate with business impact and ROI data are the following:
- expected lifecycle of the HR program;
- importance of the program to strategic objectives;
- cost of the program;
- time commitment of participants;
- visibility of the program;
- extent of management interest.
Table 6 lists some programs often taken to these levels of analysis. The table represents both impact and ROI analysis, but it is helpful to review the distinction between the two levels. Sometimes the business impact of the program is desired without the subsequent ROI analysis. For example, a program designed to improve job satisfaction may be evaluated only at the business impact level. In that case, you’ll want to develop the direct link between the HR program and job satisfaction. The ROI methodology requires that you develop the monetary benefits of the program and compare them to the costs of the program. Because of the difficulty in converting job satisfaction data to monetary value, you may not wish to pursue ROI analysis in this example.
Expected Lifecycle of the Program. The first criterion in selecting programs for business impact and ROI analysis is the length of time the program is in existence (lifecycle). Some HR programs are one-time opportunities designed to react to a particular issue or tackle a particular problem in the organization. These programs are intended to be brief, and ROI analysis may not be necessary. For example, an early retirement offer with a short deadline or mandatory ethics training for all employees may be one-shot programs. Conversely, some programs seem to exist forever. For example, a new-employee orientation program will always be needed as long as there are new employees. Consequently, at some point in its lifecycle, it may be helpful to conduct a comprehensive analysis. Nortel Networks pursued an ROI evaluation of its orientation program because it was considered a permanent fixture.
Linkage to Strategic Initiatives Objectives. Another important issue is the linkage of the program to strategic initiatives or operational goals. A strategic program is one designed to address specific strategic objectives. These programs are so important that they should be subject to a high level of scrutiny.
For example, a major customer service initiative designed to support a strategic goal of improving customer satisfaction may be a candidate for ROI. BellSouth Corporation evaluated one of its major customer service HR programs because it was linked to several strategic initiatives. Other programs may be operationally focused, adding significant impact to the organization’s bottom line. For example, workout programs, similar to those instituted at General Electric where managers tackled particular issues and problems in a formal development effort may be suitable candidates for ROI. These programs are designed to add value and, consequently, should be subjected to ROI analysis to see if they are adding appropriate value as intended. A large metropolitan transit system pursued an ROI study on an absenteeism reduction program for bus drivers because of the operational problems related to excessive absenteeism.
Cost of the Program. Expensive programs need a comprehensive level of analysis to ensure that they are adding appropriate value. For example, in a large commercial bank, an expensive leadership development program costing $100,000 per candidate was subjected to ROI analysis to show the actual value, using a sample of participants. The board of directors wanted this evaluation because of perceived excessive costs. Another example: The Royal New Zealand Navy pursued an ROI study on a retention bonus plan for marine engineers because the plan cost the Navy $4 million. In contrast, however, a tuition reimbursement program for off-the-job education programs directly related to the job utilized by only 3% of the employees may not be a candidate for ROI analysis.
Time Commitment. Programs that involve much time are also candidates for business impact and ROI analysis. If employees must take a great deal of time away from their jobs to attend meetings and learning sessions, it can be helpful to determine if the process is adding value.
For example, at Allied Irish Bank, senior executives questioned the value of a 360-degree feedback process because it required so much of the managers’ time. The managers complained about filling out the many forms, receiving feedback data, and attending training sessions and meetings to analyze and understand the process. This excessive time caused executives to question the value of the program.
Visibility of the Program. Highly visible or controversial programs often generate a need for accountability at higher levels. Because such programs can stimulate concern among their critics, they require a higher level of accountability. For example, a wellness and fitness center built for employees at a major automotive manufacturing complex was selected for ROI analysis based on its visibility. Sometimes managers and shareholders question the value of a high-profile project or program, thus elevating the evaluation to the business impact or ROI level.
Management Interest. The extent of management interest is often the most critical issue in driving programs for impact and ROI analysis. Senior executives have concerns about some programs but not all of them. Based on feedback they receive or their own perceptions of the program, they want this level of accountability applied. For example, a National Aeronautics and Space Administration (NASA) executive recently required an impact analysis for one of its education programs, the NASA Faculty Fellowship Program. Senior administrators wanted to know if the program was adding value to NASA and to the universities. The good news is that there are only a few programs where these concerns exist. Even if the other criteria do not apply, it may be helpful or even necessary to elevate the evaluation to these levels to satisfy executive concerns. So-called “soft” programs that deal with diversity, motivation, empowerment, and communication are often scrutinized closely by executives who are not clear about the programs’ value to the organization.
Return to Book.