Evaluating software capabilities

Three primary criteria were used to select programs for review. First, the program can be purchased independent of consulting services; second, it can run on a stand-alone PC; and third, it has a base of users for the product reviewed or an earlier one. Four programs meeting those criteria were selected.

CompStar Appraiser Plus 360 from Questar InfoComm of Salt Lake City and 20/20 Insight GOLD from Performance Support Systems Inc. of Newport News, Va., were near completion of beta testing when reviewed. The two systems were scheduled for release early in the fourth quarter 1998 and may be further enhanced in the future. PulseTools, introduced earlier this year by Vitality Alliance of Provo, Utah, is an upgrade and companion module to CorporatePulse. Intelligent Consensus 3.1 from Teams International of Tempe, Ariz., has been on the market since 1994. The company planned to introduce Version 3.2 during fourth quarter 1998, but the beta version was unavailable for review. Teams International is also introducing a service option, TEAMS360.com, which allows companies access to the software on a fee basis through a custom web site.

To help you assess the programs, I compared them with various criteria identified in the following three scenarios for level of need.

A basic, low-cost system. Suppose your company has 300 employees and a budget of $10,000 for this project. You recently set up a workable 360 Degree process using paper forms but now need to automate the process and track respondents to administer the system more efficiently. You want to keep things simple and avoid overwhelming employees with extensive reports. CompStar Appraiser Plus 360 (CompStar) can handle those basic needs within your budget. This solid tool packs in some powerful features for a modest price. It’s primarily designed for network use and has the option of exporting surveys for off-site evaluations. A modest library of core job-behavior questionnaire items is included, along with goal-planning and development modules. Questionnaire items can be fully customized, and you can choose from a variety of rating scales and customize the weighting descriptors. Respondents can insert comments to support their ratings of job behavior, as well as summary comments. The program permits weighting of individual job behaviors, goals and development plans; the weighting can be applied within a respondent category and by type of respondent.

The software currently generates only individual reports containing each respondent’s ratings and comments but does not provide a consolidated report. Also lacking are options for graphic display of reports, such as shaded or color-coded horizontal bars, to compare respondent ratings. The individual reports do include a final summary page by respondent category—such as self, peer or supervisor—that lists a consolidation of average feedback for overall job behaviors, goals and development.

There is a way around the reporting limitation. Because the program’s database is managed in Microsoft Access, the user can create custom reports by exporting the data and using Access’s report-writing function.

CompStar does a good job of tracking respondents but does not provide for notifying them of pending evaluations by e-mail. Also, the program’s respondent categories are limited to “self,” “supervisor” or “peer.” Other respondents, such as “customers” and “direct reports,” must be placed in the “peer” category.

“Limiting respondent categories was not a problem for our project at Sharp Electronics,” says Susan C. White, an external implementation consultant from HR Performance Solutions in Wayne, N.J. “Sharp’s application is for high-performance technical teams, so we are primarily concerned with peer feedback. Initial setup of the system was more difficult than expected, but once complete, it really hummed—and employees found it very easy to use. We like it because it offers so much flexibility for the price. However, we’d prefer to see the reporting function improved and discovered that the rating-level descriptions did not show up on the data entry screens. This required us to provide employees with an explanation of the ratings before they could enter their evaluations.”

20/20 Insight GOLD and Intelligent Consensus could perform all desired functions and gave greater flexibility in reporting and respondent options. They also have many more media distribution options, such as paper forms, diskettes and e-mail attachments through the Internet and intranets. Both systems have integrated a respondent notification feature to alert respondents of pending evaluations. But the additional features are probably overkill in this instance—and carry a higher price tag. CorporatePulse/PulseTools (CorporatePulse) could meet the price demands and deliver equally well on all criteria except one. It does not track individual respondents because it is designed to maintain ultimate confidentiality.

Going beyond the basics. In the initial stages of 360 Degree feedback, you may need help orienting your employees to the process and assistance in selecting and writing valid questions. You want an efficient, highly flexible, confidential system with many reporting options and graphic displays. You want built-in action planning, development options and suggested resources.

Three of the systems will meet most of those needs at varying levels. All four offer considerable flexibility, customization and resources for questionnaire development and reporting functions. CorporatePulse provides three days of extensive training. While participants spend part of their training on system administration, they focus primarily on writing good questionnaire items and procedures for pilot testing the questionnaires. During the last half-day of training, participants work on a “live” project and receive individual assistance with questionnaire design. The software also contains an extensive library of pretested questions. Both the training and the library items are included in the base price.

“We found the training CorporatePulse provided was excellent. It helped us customize our questions and tie them directly to our leadership academy,” says Sherri Sayre, a member of the leader enhancement and associate development team for NationsBank in Jacksonville, Fla. “We also needed media options. Not everyone in the company has computer access or is comfortable using a computer. Generating scannable forms directly from the system was a real plus for us.”

20/20 Insight GOLD also includes a broad library of well-researched questions in its base price. However, users incur a separate charge for one-and-a-half days of systems-focused training for administrators. By contrast, Intelligent Consensus has an extra charge for accessing the company’s survey library, but the base price includes two days of system administrators’ training.

Specific planning and implementation issues are extensively addressed by 20/20 Insight GOLD’s bonus module, The 360 Smart Kit. It includes information on the 360 Degree process, suggested PowerPoint presentation slides for explaining the project to employees and management and lesson plans for teaching employees how to provide appropriate performance feedback.

The Project Planner, a module of CorporatePulse, assists with project administration. The planner provides a time-line tracking function of all project steps involved in general survey development, administration and analysis. The program’s online tutorial gives additional—but limited—help with project planning and communication strategies for 360 Degree feedback. Intelligent Consensus provides comprehensive consulting services at additional cost for training design teams and for the general design process and policy development. Teams International also offers related training options, such as motivating and sustaining change, action planning and leadership coaching.

“We were confident that we could design the survey on our own based on the flexibility of Intelligent Consensus,” explains Lili Cordeiro, organizational development consultant at Westpark Hospital in Toronto. “Where we really needed help was with process; so, we engaged their consultants. In two days we had all the issues resolved, which resulted in a smooth implementation. The software increased our efficiency. Our end users love it and tell us they find it very easy. But our system administrators found the setup complex, and we needed some additional hand-holding beyond the initial training. Fortunately, their technical staff was extremely patient and accommodating—even when the error was ours!”

All three of the programs, except CorporatePulse, include varied reporting options with graphics, gap analysis and summaries. CorporatePulse cannot incorporate comments into the overall report because the system is designed as a flat file, not a relational database. Thus, a separate report is required for comments. Intelligent Consensus and 20/20 Insight GOLD are highly flexible and offer a range of reporting options to display data and integrate comments within the report.

20/20 Insight GOLD offers two features for developing action plans and providing development resources. Employees can load an Individual Development Planner diskette on their systems and track personal progress. The second feature, a comprehensive report on development recommendations, can be customized by users. It identifies low-rated items and gives detailed explanations on how to improve.

Wayne Young, employment specialist at First Technology Credit Union in Beaverton, Ore., says his review team asked many questions in evaluating products. “I think the biggest thing we did right was to take our time,” he says. “We spent considerable effort clarifying our purpose and desired outcomes before we even started looking at software. We wanted a program that offered us tremendous flexibility, could be administered twice a year but wouldn’t require a lot of back-end administrative time. With that in mind, 20/20 Insight GOLD became the clear choice.”

Intelligent Consensus also provides a powerful option for development recommendations in its Action Planning Module, available at extra cost. The module ranks all ratings from top to bottom and creates a Criteria Ranking Report. Next, it automatically selects the low-rated areas and customizes a highly specific action plan, including detailed activities and resources for employee development.

CorporatePulse lacks online action planning or development resource modules. The program does offer a proprietary instrument for measuring interpersonal communication skills, which can be adapted for use with 360 Degree feedback although it is not specifically designed for that process.

Advanced functions. Perhaps you are concerned that some participants may try to “game” the system. You want to be able to identify rater collusion and reduce evaluator bias. Finally, you need a broad view of organizational improvement over time and want to compare performance across organizational sectors or between departments.

Intelligent Consensus handles all of those functions with IntelliReporting. This special module, offered at extra cost, can provide specific reports to analyze rater collusion and rater bias. Such reports are not available with the other systems.

Intelligent Consensus also can seamlessly gather and summarize all performance data across the organization for comparison reports. Similarly, 20/20 Insight GOLD can easily provide organizational cuts of performance data through its aggregate-reporting feature. With CompStar, the need can be met only by exporting and manipulating the data in Access. CorporatePulse does not offer organization-wide reporting as a feature built into the system; the administrator must export and manipulate the data in another program to develop reports.

Other evaluation criteria
The chart provides direct comparisons of selected features, plus vendor information and technical requirements. Make sure you choose a company with a solid reputation and staying power. Ask about technical support, maintenance contracts and upgrades. Finally, call its customers. Find out why they selected the software, what they like most about it and the quality of service they received. Investing in 360 Degree software is more than merely licensing a program. It’s a significant tool designed to help you sustain the process of continuous employee development.