Questionnaire design

Scope and purpose

A questionnaire is a set of questions designed to collect information from a respondent. A questionnaire may be interviewer-administered or respondent-completed, using paper-and-pencil methods of data collection, telephone methods or computer-assisted modes of completion or Internet data collection. Questionnaires play a central role in the data collection process. They have a major impact on respondent behavior, interviewer performance, collection cost and respondent relations and therefore on data quality.

Questionnaires in ongoing surveys should be evaluated periodically. All new and modified questionnaires developed at Statistics Canada must be tested in both English and French before implementation, as required by the Agency’s Policy on the Review and Testing of Questionnaires (Statistics Canada, 2002a).

Guidelines

• Use words and concepts in questionnaires that have the same meanings for both respondents and questionnaire designers, and, in the case of businesses, choose questions, time reference periods, and response categories that are compatible with the establishment's record-keeping practices. To the extent possible, harmonize concepts and wording with those already in use. When appropriate, reuse questions from other surveys.
  
  
• Choose question design and wording that encourage respondents to complete the questionnaire as accurately as possible. To this end, the questionnaire must focus on the topic of the survey, be as brief as possible, flow smoothly from one question to the next, facilitate respondents' recall and direct them to the appropriate information source.
  
  
• In the introduction to the questionnaire, provide the title or subject of the survey, identify the sponsor, explain the purpose of the survey, and request the respondent's cooperation. Also indicate the authority under which the survey is taken, and what confidentiality protection measures, record linkage plans and any data sharing arrangements that are in place (Statistics Canada, 1998a). The opening questions should be applicable to all respondents, be easy and interesting to complete, and establish that the respondent is a member of the survey population.
  
  
• Ensure that the value of providing information is made very clear to respondents, and explain why it is important to complete the questionnaire and how the survey data will be used.
  
  
• Design self-completed questionnaires to be attractive and easy to complete. To this end, give a positive first impression in the cover letter and front cover, and make the questionnaire appear professional and businesslike. If it is to be interviewer-administered, make the questionnaire interviewer-friendly.
  
  
• To minimize the possibility of reporting errors, ensure that the instructions to respondents and/or interviewers are short, clear, and easy to find. Provide definitions at the beginning of the questionnaire or in specific questions, as required. Ensure that time reference periods and units of response are clear to the respondent, use boldface print to emphasize important items, specify "include" or "exclude" in the questions themselves (not in separate instructions), and ensure that response categories are mutually exclusive and exhaustive.
  
  
• With respect to the questionnaire layout, provide titles or headings for each section of the questionnaire, and include instructions and answer spaces that facilitate accurate answering of the questions. Use color, shading, illustrations and symbols to attract attention and guide respondents or interviewers to the parts of the questionnaire that are to be read and to indicate where answers are to be placed. At the end of the questionnaire, provide space for additional comments by respondents and include an expression of appreciation to the respondent.
  
  
• Choose among a wide range of methods to test and evaluate the questionnaire. The suitability and intensity of their use depend on various factors and circumstances. These include the type and size of the survey, the survey's content, utilization of previous survey questions or standard questions, whether it is an ongoing collection or not, the method of data collection, the project schedule, the budget, and the availability of resources (Statistics Canada, 2002a).
  
  
• Consider two or more phases of questionnaire testing. This involves testing the questionnaire at an early stage of its development, making revisions to the questionnaire based on the findings, and then testing the revised questionnaire. This process may be repeated through two, three or even more phases of testing. Different methods of testing the questionnaire may be used during each phase of testing.
  
  
• Use qualitative testing to provide insight into how respondents react to a questionnaire. Methods include focus groups and in-depth interviews, cognitive methods such as think-aloud interviews and paraphrasing, and behavior coding. Focus groups and one-on-one, in-depth interviews are used to test and evaluate question wording, sequencing and format. Cognitive methods are used to examine respondents' thought processes as they answer the survey questions and to ascertain whether or not they understand the questions and are able to provide accurate responses. Behavior coding provides a systematic and objective means of examining the effectiveness of the questionnaire by analyzing the interviewer-respondent interaction. Qualitative testing may also be used to help determine questionnaire content through the evaluation and exploration of key concepts.
  
  
• Conduct informal testing (pretesting) of the questionnaire to help identify poor question wording or ordering, errors in questionnaire layout or instructions, and problems caused by the respondent's inability or unwillingness to answer the questions. Use informal testing to suggest additional response categories that can be pre-coded on the questionnaire, and to provide a preliminary indication of the interview length and potential nonresponse problems.
  
  
• Hold debriefing sessions with interviewers after testing the questionnaire. Let interviewers discuss their experiences in interviewing respondents and how the questionnaire performed. They can identify potential sources of response and nonresponse errors as well as areas where the questionnaire can be improved.
  
  
• Use split sample testing when there is a need to determine the "best" of two or more alternative versions of the questionnaire. This involves an experimental design that is incorporated into the data collection process to investigate issues such as question wording, question sequencing, and data collection procedures
  
  
• Conduct pilot testing after a thorough questionnaire test to observe how all the survey operations, including the administration of the questionnaire, work together in practice. The pilot test duplicates the final survey design on a small scale from beginning to end, including data processing and analysis. It provides an opportunity to fine-tune the questionnaire before its use in the main survey.
  
  
• Verify French and English versions of the questionnaire for consistency.
  

Top of Page

Converse, J.M. and Presser, S. (1986). Survey Questions: Handcrafting the Standardized Questionnaire. Sage University Paper Series on Quantitative Applications in the Social Sciences, 07-063, Sage Publications, Thousand Oaks, California.

DeMaio, T.J. (ed.) (1983). Approaches to developing questionnaires. Statistical Policy Working Paper 10, United States Office of Management and Budget, Washington, D.C.

Fowler, F.J. Jr. (1995). Improving Survey Questions: Design and Evaluation. Applied Social Research Methods Series, 38, Sage Publications, Thousand Oaks, California.

Gower, A.R. (1994). Questionnaire design for business surveys, Survey Methodology, 20, 125-136.

Platek, R., Pierre-Pierre, F.K. and Stevens, P. (1985). Development and Design of Survey Questionnaires. Statistics Canada Catalogue no. 12-519E.

Schuman, H. and Presser, S. (1996). Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording and Context. Sage Publications, Thousand Oaks, California.

Statistics Canada (1998a). Policy on Informing Survey Respondents. Policy Manual, 1.1.

Statistics Canada (2002a). Policy on the Review and Testing of Questionnaires. Policy Manual, 2.8.

Sudman, S. and Bradburn, N.M. (1982). Asking Questions: A Practical Guide to Questionnaire Design. Jossey-Bass, San Francisco.

Tanur, J.M. (ed.) (1992). Questions about Questions: Inquiries into the Cognitive Bases of Surveys. Russell Sage Foundation, New York.