Workshops

• Workshop 1. The Psychology of Survey Response (Workshop is full)
• Workshop 2. Mixed-Mode Surveys: Innovation and Challenge (Workshop is full)
• Workshop 3. Using Paradata to Weight for Non-response (Workshop is full)

Three full-day workshops will take place on Tuesday October 28th. Simultaneous translation will be provided

Please note that the number of seats for each workshop is limited

Workshop 1. The Psychology of Survey Response (Workshop is full)

Roger Tourangeau, University of Maryland, USA
(English presentation with simultaneous translation and with French and English materials)

This course examines survey questions from a psychological perspective. It describes the major psychological components of the response process, including comprehension of the questions, retrieval of information from memory, combining and supplementing information from memory through judgment and inference, and the reporting of an answer. It discusses models of how respondents answer questions in surveys, reviews the relevant psychological and survey literatures, and traces out the implications of these theories and findings for survey practice, especially for the design of questionnaires. The class focuses mainly on the latter stages of the response process, discussing how respondents “format” their answers (and the characteristic pitfalls of different types of survey items), how they sometimes “edit” their answers before reporting them, and how different modes of data collection affect the response process. In addition, the course will examine cognitive interviewing, other methods for developing and testing survey questions, and other techniques that have emerged from the effort to apply cognitive theories to survey methodological problems. The course reviews both the theories and the growing empirical literature about the survey response process.

Both attitudinal and behavioural questions will be discussed. The course will include examples that applying findings from the literature on cognitive aspects of survey methods to improve the quality of survey data.

Program

Introduction and Overview

Mapping and Formatting Answers

• Formats for survey questions
• Scales and the range-frequency model
• Departures from the model
• Open-ended items and rounding
• Closed-ended items and response order effects

Editing of Answers

• Definition of sensitive questions
• Consequences of asking sensitive questions (unit non-response, item non-response, and misreporting)
• Methods for countering the effects of sensitivity (bogus pipeline, self-administration, randomized response technique, and other methods)

Mode of Data Collection

• Dimensions on which modes of data collection differ
• Visual vs. auditory channel
• Self-administration
• Interactivity and computers as social actors

Cognitive Interviewing and Other Response Aids

• Issues with cognitive interviewing—incompleteness, reactivity
• Best practices for cognitive interviews
• Other methods for pretesting
• Assessment of the virtues and limitations of the cognitive approach

Top of Page

Workshop 2. Mixed-Mode Surveys: Innovation and Challenge (Workshop is full)

Dr. Edith D. de Leeuw, Utrecht University, Netherlands
(English presentation with simultaneous translation and with French and English materials)

When planning a survey, many decisions have to be made and one of the most important decisions concerns the choice of data collection mode. At present a large variety of data collection methods are available for social surveys and official statistics, which leads to methodological questions, such as, which methods is best? Each method has its advantages and disadvantages; each method also makes different logistical demands. Sometimes the choice for a particular data collection is easy and straightforward. But often the situation is more complex and one single method will not suffice. Therefore multiple modes of data collection or mixed modes have become more and more popular in survey practice.

The topic of this workshop is the methodology for mixed-mode surveys. In this workshop, I will give an introduction and overview of methodological issues involved in the designing, implementation, and evaluation of mixed mode surveys. We will discuss advantages and disadvantages of mixed-mode survey design and review common forms of mixed-mode design, reasons for using more than one mode in a survey and its consequences. The emphasis will be on data quality and on ensuring equivalence in a mixed-mode design. This workshop draws on empirical results from mode experiments as well as practical and theoretical considerations for the design and implementation of mixed-mode surveys. The workshop does not focus on technical aspects of mixed-mode design (software, programming) nor on the mathematics for adjustment, although general principles of adjustment will be reviewed.

The objective of this workshop is to provide the participants with a theoretical background on mixed mode methodology and with an empirical knowledge base on the implications of mixed-mode for questionnaire design, total survey error and logistics. A text on mixed-mode design and a power-point handout will be available for all participants.

Program

Introduction of the Teacher and Topic

Why mix modes? Total survey error perspective

Mixed mode surveys: Some special cases

Taxonomy of multiple modes: Multiple mode contacts

Taxonomy of mixed mode surveys: Measurement error

Why modes differ: Visual vs. aural, self-administered vs. interviewer-guided

Traditional questionnaire design and its implication for mixed-mode

Designing for mixed-mode: two cases

Approaches to questionnaire design and measurement error

• Method optimization
• Unified (uni) mode design
• Generalized mode design

Logistics of multiple mode surveys

Consultation, Questions, Discussion

Suggested readings:
De Leeuw, E.D. (2005). To mix or not to mix data collection methods in surveys. JOS, Journal of Official Statistics, 21,2, 233-255 (also available on www.jos.nu)

Top of Page

Workshop 3. Using Paradata to Weight for Non-response (Workshop is full)

Jean-François Beaumont, Statistics Canada
(French presentation with simultaneous translation and with English and French materials)

The main purpose of this workshop is to introduce the basic concepts related to weighting when there is non-response in surveys, paying particular attention to the use of paradata. Paradata are data on the data collection process itself, such as the number of attempts to contact the survey’s sampled units or the day and time of these attempts. Since the introduction of computer assisted interviews, this type of data is frequently collected. Recently, people have started studying its properties to improve the collection process. In this workshop, we focus more on the use of this data for statistical purposes in handling non-response.

We will start by introducing certain basic concepts regarding non-response to properly describe the links between data collection, non-response and weighting. We will also compare imputation and weighting for non-response as methods for handling missing data. Then we will briefly discuss weighting in the ideal situation where there is no non-response. The main part of the workshop will cover the different techniques used to adjust the weights to compensate for non-response, such as the use of logistic regression and the score method. We will discuss the essential conditions for using auxiliary data, such as paradata, which can be used to control bias and variance due to non-response. Finally, we will briefly describe a few techniques for estimating variance in the presence of non-response, such as the Taylor series linearization and resampling methods such as the bootstrap.

Program

Introduction

• Objectives and outline of workshop
• Situating data collection, non-response and weighting in the survey process
• Types of error (sampling error and non-sampling errors)
• Definitions
  • Partial non-response vs. total non-response
  • MAR (Missing At Random) vs. NMAR (Not Missing At Random)
• Edit and imputation
• Main causes of non-response (refusal, non-contact, etc.)
• Prevention of non-response

Weighting and inference under the sample plan (when there is no non-response)

• Sampling theory in surveys
• Common estimators
  • Horvitz-Thompson estimator
  • Calibration estimator

Methods of processing and inference where there is non-response

• Relationship between two-phase sampling and non-response
• Under-sampling of non-respondents
• Common options for treating non-response
  • Reweighting
  • Imputation

Weighting for non-response

• Response probabilities modelling approach
  • General concepts
  • Choice of auxiliary variables for estimating response probabilities, potentially including variables on the collection process
  • Methods of estimating response probabilities
    • Logistic regression
    • Creation of reweighting classes (scores method and CHAID)
    • Non-parametric regression
    • Survey weights - to use or not to use?
  • Smoothing of weight adjustments
  • Does response probability depend on variables of interest?
    Calibration
  • Illustration using a simple artificial example
  • Illustration using the Canadian Labour Force Survey
• Approach entirely based on calibration without explicit modelling of response probabilities

Estimation of variance

• Variance and estimation of variance when there is no non-response
  • Taylor series linearization
  • Resampling methods (Jackknife and Bootstrap)
• Variance and estimation of variance where there is non-response
  • Small sampling fraction vs. large sampling fraction
    • Taylor series linearization
    • Resampling methods (Jackknife and Bootstrap)