Questionnaire design

Survey objectives and data requirements
Analysis plan
Survey target population
Method of data collection
Size of the survey
Data processing plans
Budget
Time
Questionnaire testing
Data quality
Abbreviations and acronyms
Complex words and terminology
Frame of reference
Specific questions
Double-barreled questions
Loaded questions
Open or closed questions

Questionnaires play a central role in the data collection process. A well-designed questionnaire efficiently collects the required data with a minimum number of errors. It facilitates the coding and capture of data and it leads to an overall reduction in the cost and time associated with data collection and processing. The biggest challenge in developing a questionnaire is to translate the objectives of the data collection process into a well-conceptualized and methodologically sound study.

The following questions should be addressed:

• Why is this survey being conducted?
• What do I need to know?
• How will the information be used?
• How accurate and timely does the information have to be?

Before designing the questionnaire, many decisions have to be made. These decisions affect the questionnaire, and should be part of the draft plan for a survey. The draft plan should address the following issues:

Survey objectives and data requirements

In order to address the survey's objectives, you should prepare a document that provides a clear and comprehensive statement of the survey's goals, data requirements, and the analysis plan. This document will determine the variables to be measured, and ultimately, the survey questions and response alternatives. When formulating the questions, consult with subject-matter experts and if possible, members of the target audience. Also, examine questions from other surveys on the same or similar topics. This research will provide you with a useful starting point and will help you create appropriate and informative questions. Make certain that the questions are relevant to the survey objectives and information requirements and ensure that there is an established rationale behind each question. Also, you should explain how the information gathered from these questions will be used and whether they will be good measures of the required data.

Analysis plan

The next step in designing a questionnaire is to create an analysis plan. First, outline the questionnaire's objectives and data requirements. Describe the target audience as clearly as possible. Then, identify the reference period (the time period under construction—in the last year, in the last month etc.). Develop a list of the units to be sampled (e.g., students, houses, teachers, etc.). Decide on the method of data collection to be used (e.g., face-to-face interview, telephone interview, mailed questionnaire, etc.). Explain how the questionnaire content, wording, format and pre-testing process will be developed; as well as the procedures put in place to deal with the interviewer training and non-response results. Also, choose the methods to be used during the data processing (e.g., coding, editing etc.). Some of the other issues that can be analysed during this step include estimation methods, result output tabulations, result reports and the analysis. Finally, the last two important issues to be considered are the time required to complete the entire process and the budget that has been allotted to it.

Survey target population

Often the target population (the population for which information is required) and the survey population (the population actually covered) differ for practical reasons, even though they should, in actuality, be the same. Sometimes, it is necessary to impose geographical limitations excluding certain parts of the target population because they are inaccessible due to difficulty or cost.

It is also possible that some of the survey concepts and methods that are used can be considered inappropriate for certain parts of the population. For example, consider a survey of post-secondary graduates where the objective is to determine if the graduates found jobs and, if so, what types of jobs. In this case, you might exclude graduates who specialized in religious seminaries or military schools, as these types of graduates would be reasonably assured of securing employment in their respective fields. Thus, the target population might contain only those graduates who graduated from universities, colleges and trade schools.

Method of data collection

This next step in questionnaire design involves developing the methods of data collection. This is important step because you need to consider the costs, physical resources, and time required to conduct the survey.

First, select the best method for gathering the required data. Keep in mind that cost and data quality will be directly impacted by the method you choose.

There are several options available: face-to-face interviews or computer assisted personal interviewing (CAPI) are two examples. These methods are administered by a trained interviewer and can have either a structured or unstructured line of questioning. There are also two telephone methods available: telephone interviews or computer assisted telephone interviewing (CATI). Both of these methods are also administered by a trained interviewer, but the telephone versions are structured with a more formal interview schedule. Finally, there is also the option of a collecting data through a self-completed questionnaire. This method allows the respondent to complete the questionnaire without the aid of an interviewer. It is highly structured and can be returned by mail or through a drop-off system.

Size of the survey

Since each survey is different, there are no hard and fast rules for determining its size. The deciding factors in the scale of the survey operations are time, cost, operational constraints and the desired precision of the results. Evaluate and assess each of these issues and you will be in a better position to decide the sample size. Also, consider what should be the acceptable level of error in the sample. If there is a lot of variability in the population, the sample size will need to be bigger to obtain the specified level of reliability.

Data processing plans

This processes the questionnaire responses into output. Coding; data capture; editing; dealing with invalid or missing data; and, if necessary creating derived variables are the tasks that will be completed during data processing. In short, the aim in this step is to produce a file of data that is as free of errors as possible.

Budget

Sometimes, questionnaire design is decided upon by the amount of money available to do a specific survey. Costs are one of the main justifications for choosing to conduct sample surveys instead of a census. With surveys, it is possible to obtain reasonable results with a relatively small sample or target population. For example, if you need information on all Canadian citizens over 15 years of age, a survey of a small percentage of these (1,000 or 2,000 depending on the requirements) might provide adequate results.

Time

One of the advantages of survey sampling is that it permits investigators to produce the information quickly. It is often the case that survey results are required shortly after the need for information has been identified. For example, if an organization wants to conduct a survey to measure the public awareness of a media advertisement campaign, the survey should be conducted shortly after the campaign is undertaken. Since sampling requires a smaller scale of operation, it reduces the data collection and processing time, while allowing for greater design time and more complex processing programs.

Questionnaire testing

This is a fundamental step in developing a questionnaire. Testing helps discover poor wording or ordering of questions; identify errors in the questionnaire layout and instructions; determine problems caused by the respondent's inability or unwillingness to answer the questions; suggest additional response categories that can be pre-coded on the questionnaire; and provide a preliminary indication of the length of the interview and any refusal problems. Testing can include the entire questionnaire or only a particular portion of it. A questionnaire will at some point in time have to be fully tested.

Data quality

This step identifies errors and verifies results. No matter how much planning and testing goes into a survey, something unexpected will often happen. As a result, no survey is ever perfect. Quality assurance programs such as interview training, information editing, computer program testing, non-respondent follow-ups, and data collection and output spot-checks are required to minimize non-sampling errors introduced during various stages of the survey. Statistical quality-control programs ensure that the specified error levels are controlled to minimum.

There is a lot to consider when developing a questionnaire. The following is a list of some key points to think about when designing your questionnaire:

• Is the introduction informative? Does it stimulate respondent interest?
• Are the words simple, direct and familiar to all respondents?
• Do the questions read well? Did the overall questionnaire flow?
• Are the questions clear and as specific as possible?
• Does the questionnaire begin with easy and interesting questions?
• Does the question specify a time reference?
• Are any of the questions double-barreled?
• Are any questions leading or loaded?
• Should the questions be open- or close-ended? If the questions are close-ended are the response categories mutually exclusive and exhaustive?
• Are the questions applicable to all respondents?

The introduction of the questionnaire is very important because it outlines the pertinent information about the survey being conducted. The introduction should:

• provide the title or subject of the survey;
• identify the sponsor;
• explain the purpose of the survey;
• request the respondent's co-operation; and
• inform the respondent about confidentiality issues, the status of the survey (voluntary or mandatory) and any existing data-sharing agreements with other organizations.

Respondents frequently question the value of the gathered information to themselves and to others. Therefore, be sure to explain why it is important to complete the questionnaire, how the information will be used, and how respondents can access the results. Ensuring that respondents understand the value of their information is vital in undertaking a survey.

The opening questions of any survey questionnaire should establish the respondents' confidence in their ability to answer the remaining questions. If necessary, the opening questions should help determine the respondent as a member of the survey population.

One of the most important factors in any survey is the design of the actual questionnaire. The questions and instructions need to be easy to understand and respond to. The way a question is worded is also very important. The same question worded in a different manner may achieve completely different results. Consider the following:

Abbreviations and acronyms

Complex words and terminology

Frame of reference

Specific questions

Double-barreled questions

Each of the above examples asks two questions rather than one:

In the first example, the question asks respondents what language they first learned in childhood, and what language they still understand.

In the second example, the question asks respondents if they plan to leave their cars at home, and whether or not they are taking the bus for the next year.

The third example asks respondents if their company provides training for new employees as well as providing retraining for existing employees.

In some instances, the response for each half of the question is the same. However, many other responses could include two very separate answers which would make interpreting this question difficult.

Loaded questions

The following examples demonstrate how a loaded question can impact the respondent's results.

The wording of the first question asks whether the respondents were in favour of Sunday shopping, while the second question was worded to ask respondents whether they were in favour of not working on Sundays. As a result, there was a significant change in the data.

A possible explanation for the difference in the results could be that some respondents did not quite understand the implications of the question. Some people may be opposed to working on Sundays, but are still in favour of shopping. However, if no one works on Sundays, then stores cannot stay open for shoppers!

Open or closed questions

Generally there are two types of questions: open and closed. Open questions give respondents an opportunity to answer the question in their own words. Closed questions give respondents a choice of answers and the respondent is supposed to select one.

There are advantages and disadvantages to using one type of question versus another. The open question allows the respondent to interpret the question and answer it anyway he or she chooses. The respondent writes the answer or the interviewer records verbatim what the respondent says in answer to the question.

The closed question restricts the respondent to select an answer from the specified response options. For the respondent, a closed question is easier and faster to answer and for the researcher, closed questions are easier and less expensive to code and analyse. Also, closed questions provide consistency, an element that is not necessarily going to occur with an open question.

A good questionnaire ends with a comments section that allows the respondent to record any other issues not covered by the questionnaire. This is one way of avoiding any frustration on the part of the respondent, as well as allowing them to express any thoughts, questions or concerns they might have. Lastly, there should be a message at the end thanking the respondents for their time and patience in completing the questionnaire.