Avoid Problems in Questionnaire Design by Thinking Like a Survey Respondent

By Lindsay Northon May 31, 2016
LIKE SAVE PRINT
Reuse Permissions

Too often, when talking about the competency of Critical Evaluation, our thinking fast-forwards to processes—interpreting information, solving problems, making decisions. Although it is wise to reflect on evaluation processes, it is also important to take a step back and consider what goes on beforehand. Think about your research methodology. How will you collect the data that is to be evaluated? Especially when using a survey, you will want to ensure that the data you collect is credible. This is true whether you are early in your career and working on your first survey or an executive overseeing the survey data collection of others.

While surveys are a great way to collect quantitative (and sometimes qualitative) information, they need to be designed so that errors in measurement are reduced. A key objective is to anticipate any cognitive issues that may arise among respondents taking the survey. Writing clear, direct questions and providing all possible answer choices is critical.

The following brief questionnaire is illustrative. Read through the five questions below and see if you can detect the cognitive issues that a respondent may have when taking this survey.

  1.  How confident and motivated are you in meeting your 2016 work goals?
    1. Very
    2. Somewhat
    3. Not at all

      ISSUE: This double-barreled question asks the respondent about two different subjects: confidence level and motivation.
      AVOID IT: Break into two separate questions. Easy fix.

  2.  Do you feel that you have autonomy in your current job role?
    1. Yes
    2. No

      ISSUE: Some respondents may not understand the meaning of “autonomy.” Provide a definition of the word somewhere in the question stem or elsewhere in the survey. If you are unable to do so, add another answer choice (e.g., “Not sure” or "Not applicable).
      AVOID IT: Make sure your questions are clear by testing them with someone from the target population. Or try the cognitive interview technique if you are familiar with it.

  3. How many meetings have you attended this month?
    1. 0-5
    2. 5-10
    3. 10-15
    4. 15 or more
      ISSUES: First, the response options are not mutually exclusive. That is, a respondent who attended 10 meetings has to decide between options (b) or (c) and could choose either. Second, a respondent may find it burdensome to remember the number of meetings attended.

      AVOID IT: To address the first issue, make sure answer choices do not overlap (i.e., the above options can be changed to 0-4, 5-9, 10-14, etc.). As for the second issue, use a reasonable time frame when asking a respondent to recall an event. If you’re not sure about what might be considered reasonable, seek input from an expert reviewer or someone from the target population.

  4. Which of the following describes your ideal organizational culture?
    1. Collaborative
    2. Predictable
    3. Pragmatic
    4. Results-oriented

      ISSUE: Many respondents may not find the answer they want, or may want to select more than one option.
      AVOID IT: An open-ended question may be more advantageous here. If you would prefer to keep it multiple choice, enlist the help of a mini focus group before you create the survey to gather all possible answers in the correct lingo.

  5. Which of the following best describes the competency that you do not consider an area you need to develop?
    ISSUE: One need not even read the answer choices to recognize that this question is ambiguous and negative. A respondent would not understand what is being asked—areas they need to develop, or areas they don’t need to develop? Complex, negatively worded questions cause respondents to be frustrated with the survey overall, and the more often such questions appear, the more frustrated respondents will become.
    AVOID IT: In general, avoid negatively worded questions. If you must include them, avoid making them complex: Keep the questions simple and direct. Here, are respondents really being asked what competency they need to improve in? If so, ask exactly that.

The issues discussed above are not all-inclusive. But following these simple tips to avoid respondent response challenges will help you create effective, high-quality surveys that collect strong, credible data. This will lead to better interpretations of information, solutions to problems and decisions made—in other words, better critical evaluations.

Lindsay Northon is HR competencies specialist in the SHRM Research Department.

LIKE SAVE PRINT
Reuse Permissions

MEMBERSHIP

Become a SHRM Member

Join/Renew Today

Job Finder

Find an HR Job Near You

SPONSOR OFFERS

Find the Right Vendor for Your HR Needs

SHRM’s HR Vendor Directory contains over 3,200 companies

Search & Connect