Response and nonresponse

Scope and purpose

Despite the best efforts of survey managers and operations staff to maximize response, some nonresponse will occur. For a unit to be classified as responding, the degree of item response or partial response (where an accurate response is obtained for only some of the data items required from a respondent) must meet a minimum threshold level below which the response would be rejected and considered a unit nonresponse. In such an instance, the sampled person, household, business, institution, farm or other unit is classified as not having responded at all.

Nonresponse has two effects on data: first, it introduces bias in estimates when nonrespondents differ from respondents in the characteristics measured; and second, it contributes to an increase in the sampling variance of estimates because the effective sample size is reduced from that originally sought.

Principles

The degree to which efforts are made to get a response from a nonrespondent is based on budget and time constraints, its impact on the overall quality and the risk of nonresponse bias. If nonresponse persists, adjustments are subsequently made to the data to compensate for nonresponse. Decisions on the appropriate degree of research to be undertaken to develop nonresponse adjustment techniques are likewise influenced by issues of budget, time, use of data and risk of bias. Nonresponse is monitored for feedback to survey staff for immediate and future action and is reported to users of the survey data. An effective respondent relations program and a well-designed questionnaire are critical elements in maximizing response (see section on Questionnaire design).

• A good response rate is obtained in part by ensuring an appropriate level of quality during all of the survey planning and implementation steps. Take an integrated approach so that nonresponse management techniques are not duplicated. To attain a desired response rate, keep in mind the following factors:
  
  • the quality of the survey frame (in terms of population coverage and facility of establishing contact with the respondent);
  • survey population;
    
  • method of data collection (for example, by mail, personal interview, telephone interview, computer assisted interview);
    
  • sampling method;
    
  • time of year and length of collection period;
    
  • response burden imposed (length of interview, difficulty of subject matter, periodicity of the survey);
    
  • nature of subject matter (sensitive subjects);
    
  • length and complexity of the questionnaire;
    
  • the effectiveness and scope of the follow-up methodology;
    
  • expected difficulties in tracing respondents who have moved;
    
  • prior experience with same type of survey;
    
  • prior experience and demonstrated ability of collection staff;
    
  • workload of collection staff;
    
  • established relationships with respondents;
    
  • the communications strategy;
    
  • the total budget;
    
  • allocation of the budget to the various operations;
    
  • language of the questionnaire;
    
  • the cultural backgrounds of respondents;
    
  • the importance of the survey to users and respondents;
    
  • factors related to interviewers themselves such as training, experience, interpersonal skills, rapport building and turnover; and
    
  • the use and effectiveness of respondent incentives.
    
    
• Use a pretest as well as previous occasions of the same or similar surveys, among other means, to establish an expected response rate.
  
  
• When operational constraints permit, follow up the nonrespondents (all or a sub-sample of them). Following up nonrespondents increases the response rate and can help ascertain whether respondents and nonrespondents are similar in the characteristics measured. Such follow-up is particularly important in the case of longitudinal surveys where the investment is clearly more long-term and the sample is subject to accumulating attrition (and possibly bias) due to nonresponse at each survey occasion. In this case, tracing activities are of particular importance.
  
  
• For longitudinal surveys, facilitate high quality tracing. Obtain extra contact information for sampled units at each survey occasion. Provide a “Change of address” card and ask the sampled unit to notify the Agency if a move happens in between survey cycles. This will help obtain up-to-date contact information. In addition, administrative data, city and telephone directories, and many other sources including local knowledge are valuable to the tracing staff.
  
  
• Prioritize follow-up activities. For example, in business surveys, follow-up large or influential units first, possibly at the risk of missing smaller units (see section on Editing). Likewise, give a higher priority to nonresponding units in domains with high potential for nonresponse bias. A score function can be used to prioritize the follow-up.
  
  
• Record and monitor the reasons for nonresponse (e.g., refusal, non-contact, temporarily absent, technical problem). The degree of nonresponse bias may differ depending on the reason. Monitor nonresponse trends by reason.
  
  
• Since differences between respondents and nonrespondents can cause biases in the estimates, try to determine if such differences exist. Although difficult to determine, this can be done in part by linking to external data sources (for example, administrative data files), and in part by examining the responses of the nonrespondents who were converted during a follow-up. Often it is easier to compare known characteristics of respondents and nonrespondents to see the extent of differences. In the case of longitudinal (or rotating) surveys, known characteristics of respondents at one wave of the survey can be analyzed to compare characteristics of respondents and nonrespondents at a subsequent wave. Information so gained may influence methods of compensation for nonresponse.
  
  
• Two main approaches to dealing with missing data are (Kalton and Kasprzyk, 1986): by means of sampling weight adjustment (see section on Estimation), or through the use of imputation (see section on Imputation). When appropriate, attempt to evaluate the extent to which the procedures correct for the potential bias. Take nonresponse into account when producing estimates and their associated variance estimates.
  
  
• Report response and nonresponse rates (Statistics Canada, 2000d). At Statistics Canada, standards and guidelines for reporting nonresponse have been established (Statistics Canada, 2001d). Inform users regularly of the nonresponse rate when providing estimates. Record unweighted and weighted nonresponse rates at the estimation stage on the Integrated Metadatabase (IMDB). Attempt to conform to the nonresponse reporting standard in order to facilitate comparability between surveys. The guidelines state that all units are to be classified as responding or nonresponding. Indicate clearly when there are units that responded partially, and how these units were classified.

Top of Page

Couper, M.P. and Groves, R.M. (1992). The role of the interviewer in survey participation. Survey Methodology, 18, 263-277.

Federal Committee on Statistical Methodology (2001). Measuring and reporting sources of error in surveys. Statistical Policy Working Paper 31. See also http://www.fcsm.gov.

Groves, R.M., Dillman, D. A., Eltinge, J. L. and Little, R. J. A. (2002). Survey Nonresponse. Wiley, New York.

Kalton, G. and Kasprzyk, D. (1986). The treatment of missing survey data. Survey Methodology, 12, 1-16.

Latouche, M. and Michaud, S. (1997). Concerns pertaining to weighting of longitudinal surveys. Proceedings of the Section on Government Statistics and Section on Social Statistics, American Statistical Association, 111-119.

Lévesque, I. and Franklin, S. (2000). Longitudinal and cross-sectional weighting of the Survey of Labour and Income Dynamics: 1997 Reference Year. Statistics Canada Working Paper Catalogue no. 75F0002MIE - 00004.

Statistics Canada (2000d). Policy on Informing Users of Data Quality and Methodology. Policy Manual, 2.3.

Statistics Canada (2001d). Standards and Guidelines for Reporting of Nonresponse Rates. Statistics Canada technical report.

Swain, L. and Dolson, D. (1997). Current issues in household survey nonresponse at Statistics Canada. Statistics in Transition, 3, 439-468.

Tambay, J. L., Schiopu-Kratina, I., Mayda, J., Stukel, D. and Nadon, S. (1998). Treatment of nonresponse in cycle two of the National Population Health Survey. Survey Methodology, 24, 147-156.