1. Introduction
Brady T. West and Michael R. Elliott
Between-interviewer variance in survey methodology (e.g., West, Kreuter and Jaenichen 2013; West and Olson 2010; Gabler and Lahiri 2009; O’Muircheartaigh and Campanelli 1998; Biemer and Trewin 1997; Kish 1962) occurs when survey responses nested within interviewers are more similar than responses collected from different interviewers. Between-interviewer variance can increase the variance of survey estimates of means, and may arise due to correlated response deviations introduced by an interviewer (e.g., Biemer and Trewin 1997), given the complexity of survey questions (e.g., Collins and Butcher 1982) or interactions between the interviewer and the respondent (e.g., Mangione, Fowler and Louis 1992), or nonresponse error variance among interviewers (West et al. 2013; Lynn, Kaminska and Goldstein 2011; West and Olson 2010).
Survey research organizations train interviewers to eliminate this component of variance in survey estimates, as it is sometimes larger than the component of variance due to cluster sampling (Schnell and Kreuter 2005). In reality, an interviewer variance component can never be equal to 0 (which would imply that means on the variable of interest are identical across interviewers), but survey managers aim to minimize this component via specialized interviewer training. For example, interviewers may practice the administration of selected questions under the direct supervision of training staff, and then receive feedback on any variance in administration that is noted by the staff (in an effort to standardize the administration; see Fowler and Mangione 1990). In some non-interpenetrated designs, where interviewers are generally assigned to work exclusively in a single primary sampling area (e.g., the U.S. National Survey of Family Growth; see Lepkowski, Mosher, Davis, Groves and Van Hoewyk 2010), interviewer effects and area effects are confounded, preventing estimation of the variance in survey estimates that is uniquely attributable to the interviewers. Elegant interpenetrated sample designs (Mahalanobis 1946) enable interviewers to work in multiple sampling areas, and in these cases, cross-classified multilevel models can be used to estimate the components of variance due to interviewers and areas (e.g., Durrant, Groves, Staetsky and Steele 2010; Gabler and Lahiri 2009; Schnell and Kreuter 2005; O’Muircheartaigh and Campanelli 1999; O’Muircheartaigh and Campanelli 1998).
In general, estimating the overall magnitude of interviewer variance in the measures of a given survey variable or data collection process outcome is a useful exercise for survey practitioners. If random subsamples of sample units are assigned to interviewers following an interpenetrated design, one can estimate the component of variance due to interviewers and subsequently the unique effects of interviewers on the variance of an estimated survey mean (e.g., Groves 2004, p. 364). Large estimates can indicate potential measurement difficulties that certain interviewers are experiencing, or possible differential success in recruiting particular types of sampled units. Given a relatively large estimate of an interviewer variance component and an appropriate statistical test indicating that the component is significantly larger than zero (or “non-negligible”, given that variance components technically cannot be exactly equal to zero; see Zhang and Lin 2010), survey managers can use various methods to compute predictions of the random effects associated with individual interviewers, and identify interviewers who may be struggling with particular aspects of the data collection process.
While the estimation of interviewer variance components and subsequent adjustments to interviewer training and data collection protocols have a long history in the survey methodology literature (see Schaeffer, Dykema and Maynard 2010 for a recent review), no studies in survey methodology to date have examined the alternative approaches that are available to survey researchers for comparing variance components in two independent groups of survey interviewers. In general, alternative statistical approaches are available for estimating interviewer variance components, and estimates (and corresponding inferences about the variance components) may be sensitive to the estimation methods that a survey researcher employs. The same is true for survey researchers who may desire to compare the variance components associated with different groups of interviewers, for various reasons (e.g., identifying groups that need more training or more optimal modes for certain types of questions): different statistical approaches to performing these kinds of comparisons exist, and inferences about the differences may be sensitive to the approach used. With this paper, we aim to evaluate alternative frequentist and Bayesian approaches to making inference about the differences in variance components between two independent groups of survey interviewers, and provide practical guidance to survey researchers interested in this type of analysis.
The paper is structured as follows. In Section 2, we introduce the general modeling framework that enables these comparisons of interviewer variance components for both normal and non-normal (e.g., binary, count) survey variables, and review existing literature comparing the frequentist and Bayesian approaches to estimation and inference, highlighting the advantages and disadvantages of each approach. We then present a simulation study in Section 3, evaluating the ability of the two approaches to efficiently estimate differences in variance components between two hypothetical groups of interviewers. Section 4 applies the two approaches to real survey data collected in the U.S. National Survey of Family Growth (NSFG) (Lepkowski et al. 2010; Groves, Mosher, Lepkowski and Kirgis 2009). Finally, Section 5 offers concluding thoughts, suggestions for practitioners, and directions for future research. We include SAS, R, and WinBUGS code that readers can use to implement the two approaches in the Appendix.
- Date modified: