5 Discussion
Barry Schouten, Melania Calinescu and Annemieke Luiten
This paper describes survey designs in which different population units receive different treatments or survey strategies. Differences between population units are reflected by covariates from either linked data from registrations or paradata. Survey strategies are defined as different specifications of survey design features. Such designs are termed adaptive survey designs as they adapt or tailor data collection to the population of interest. Basic ingredients of adaptive survey designs are survey strategies, population covariates, response propensities, cost and quality functions and strategy allocation probabilities. Adaptive survey designs attempt to optimize response quality by assigning different strategies to different population units. The strategy allocation probabilities represent the decision variables in the optimization.
We believe this paper contributes to the literature in three ways: it presents a general framework, it explicitly opts to choose from a set of strategies in making a quality-cost trade off, and it focuses on indicators for nonresponse error. The last two components can be found in the survey literature; it is the generalization to multiple design features and nonresponse error that is new. In its most modest form, adaptive survey designs are a stratified allocation of survey strategies over different population subgroups. In its most ambitious form, adaptive survey designs are extensions of sampling designs to multiple strategies and with a focus on nonresponse error. However, even in its most modest form, adaptive survey designs may include survey modes, incentives, reminders, length of fieldwork in face-to-face surveys, interviewer assignment and type of reporting.
Adaptive survey designs lend themselves best to settings where surveys are run repeatedly for a longer time period. In such settings, the historic information is strongest. The designs also lend themselves to survey institutes that conduct many surveys that are relatively similar in topics and budget. New and one-time only surveys ask for modesty and caution. However, this would also be true for single strategy designs. Adaptive survey designs may account for the lack of strong historic data by allowing for uncertainty in response propensities and other parameters, and by introducing a learning period or initial design phase.
In our view, the focus on nonresponse error is an important part of the framework. In this paper, we aim at representativeness of response. This aim comes from our conviction that nonresponse is always not-missing-at-random. We see larger deviations from missing-completely-at-random mechanisms for relevant auxiliary variables as indications of stronger not-missing-at-random nonresponse on survey variables given these auxiliary variables. Theoretically, this does not have to be true. Consider a simple binary yes-no survey question and 50% nonresponse. The extreme cases arise when all nonrespondents would say either yes or no. They can do so regardless of the choice of auxiliary variables and, hence, the maximal nonresponse bias on this question is the same for whatever choice of auxiliary variables. Hence, research should provide empirical support for the focus on indirect measures for nonresponse error.
Future research into adaptive and responsive designs is also needed for other questions. Research should extend designs to multiple survey errors and should investigate the robustness of designs for misspecification of models for response propensities. Until now, adaptive and responsive survey designs have focused on the nonresponse error and ignored the response or measurement error. It is well known, however, that some survey design features, e.g., survey mode or interviewers, may have a strong impact on the response error and, consequently, on the total survey error. Adaptive survey designs should, therefore, account for measurement error as well, when it can be expected that design features have a strong differential impact on response error. Optimization accounting for multiple errors represents an important area of future research.
Adaptive survey designs should in all cases be modest in the number of strategies employed in order to avoid an overly complex survey process and optimization on propensities and cost functions that are subject to uncertainty. Nevertheless, a structured way of looking is always to be preferred; adaptive designs provide such a framework and accommodate a structured search for enhanced survey designs.
Acknowledgements
The authors thank James Wagner, Mick Couper, Fannie Cobben and Mariëtte Vosmer for their useful comments on the draft version of this paper. The authors also thank the associate editor and referees for their useful remarks which have greatly improved the paper.
References
Barón, J.D., Breunig, R.V., Cobb-Clark, D., Gørgens, T. and Sartbayeva, A. (2009). Does the effect of incentive payments on survey response rates differ by income support history. Journal of Official Statistics, 25, 483-507.
Beaumont, J.-F., and Haziza, D. (2011). A theoretical framework for adaptive collection designs. Paper presented at 5th International Total Survey Error Workshop, June 21-23, Quebec, Canada.
Calinescu, M., Schouten, B. and Bhulai, S. (2012). Adaptive survey designs that minimize nonresponse and measurement risk, Discussion paper, Statistics Netherlands. Available at through www.cbs.nl.
Cobben, F. (2009). Nonresponse in sample surveys. Methods for analysis and adjustment, Ph.D. Thesis, University of Amsterdam.
De Leeuw, E.D. (2008). Choosing the Method of Data Collection. In International Handbook of Survey Methodology, (Eds., E.D. De Leeuw, J.J. Hox and D.A. Dillman), Larence Erlbaum Associates, New York, U.S.A., 113-135.
De Leeuw, E., Callegaro, M., Hox, J., Korendijk, E. and Lensvelt-Mulders, G. (2007). The influence of advance letters on response in telephone surveys. Public Opinion Quarterly, 71, 413-443.
Dillman, D.A. (2007). Mail and internet surveys: The tailored design method, Wiley, Hoboken, New Jersey, U.S.A.
Greenberg, B.S., and Stokes, S.L. (1990). Developing an optimal call scheduling strategy for a telephone survey. Journal of Official Statistics, 6, 421-435.
Groves, R.M. (1989). Survey errors and survey costs, Wiley, New York, U.S.A.
Groves, R.M., and Couper, M.P. (1998). Nonresponse in Household Interview Surveys, Wiley series in probability and statistics, Survey methodology section.
Groves, R.M., Dillman, D., Eltinge, J. and Little, R. (2002). Survey Nonresponse, New York: Wiley Series in Probability and Statistics.
Groves, R.M., and Heeringa, S.G. (2006). Responsive design for household surveys: Tools for actively controlling survey errors and costs. Journal of the Royal Statistical Society, Series A, 169, 439-457.
Hartley, H.O., and Monroe, H. (1979). Interviewer assignments which minimize the effects of nonsampling errors. Proceedings of the Survey Research Methods Section, American Statistical Association, 216-220.
Heyd, J.M., and Carlin, B.P. (1999). Adaptive design improvements in the continual reassessment method for Phase I studies. Statistics in Medicine, 18, 1307-1321.
Kalsbeek, W.D., Botman, S.L., Massey, J.T. and Liu, P.W. (1994). Cost-efficiency and the number of allowable call attempts in the National Health Interview Survey. Journal of Official Statistics, 10, 133-152.
Kersten, H.M.P., and Bethlehem, J.G. (1984). Exploring and reducing the non-response bias by asking the basic question, BPA 627-84-M1, CBS, Voorburg.
Kulka, R.A., and Weeks, M.F. (1988). Towards the development of optimal calling protocols for telephone surveys: A conditional probabilities approach. Journal of Official Statistics, 4, 319-322.
Laflamme, F., and Karaganis, M. (2010). Implementation of responsive collection design for CATI surveys at Statistics Canada, Paper presented at Q2010, 3-6 May, Helsinki, Finland.
Luiten, A., and Schouten, B. (2013). Adaptive fieldwork design to increase representative household survey response. Journal of the Royal Statistical Society, Series A, 176, 1, 169-190.
Luiten, A., and Wetzels, W. (2009). Indicators and data collection control. Work plan and preliminary findings pilot Statistics Netherlands, RISQ deliverable 8.2, www.risq-project.eu.
Lyberg, L.E., Biemer, P., Collins, M., de Leeuw, E.D., Dippo, C., Schwarz, N. and Trewin, D. (1997). Survey measurement and process quality, Wiley Series in Probability and Statistics, Wiley, Hoboken, New Jersey, U.S.A.
Lynn, P. (2003). PEDAKSI: Methodology for Collecting Data about Survey Non-Respondents. Quality and Quantity, Vol. 37, No. 3, 239-261.
Mohl, C., and Laflamme, F. (2007). Research and responsive design options for survey data collection at Statistics Canada. Proceedings of ASA Joint Statistical Meeting, Section 293, July 29 - August 2, Salt Lake City, U.S.A.
Moore, J. (1988). Self/Proxy response status and survey response quality. A review of literature. Journal of Official Statistics, 4, 2, 155-172.
Murphy, S.A. (2003). Optimal dynamic treatment regimes. Journal of the Royal Statistical Society, Series B, 65, 331-355.
Peytchev, A., Riley, S., Rosen, J., Murphy, J. and Lindblad, M. (2010). Reduction of Nonresponse Bias in Surveys through Case Prioritization, Survey Research Methods, 4, 1, 21-29.
Phillips, O., and Tabuchi, T. (2009). Responsive design for the survey of labour and income dynamics. Proceedings: Symposium 2009, Longitudinal Surveys: from design to analysis, Statistics Canada.
Särndal, C.-E. (2011a). Three factors to signal non-response bias with applications to categorical auxiliary variables. International Statistical Review, 79, 2, 233-254.
Särndal, C.-E. (2011b). The 2010 Morris Hansen Lecture: Dealing with survey nonresponse in data collection, in estimation. Journal of Official Statistics, 27, 1, 1-21.
Särndal, C.-E., and Lundström, S. (2010). Design for estimation: Identifying auxiliary vectors to reduce nonresponse bias. Survey Methodology, 36, 2, 131-144.
Schouten, B., Cobben, F. and Bethlehem, J. (2009). Indicators for the representativeness of survey response. Survey Methodology, 35, 1, 101-113.
Schouten, B., Luiten, A., Loosveldt, G., Beullens, K. and Kleven, Ø. (2010). Monitoring and changing data collection through R-indicators and partial R-indicators, RISQ Working paper. Available at www.risq-project.eu.
Schouten, J.G., Shlomo, N. and Skinner, C. (2011). Indicators for monitoring and improving representativeness of response. Journal of Official Statistics, 27, 2, 231-253.
Wagner, J. (2008). Adaptive survey design to reduce nonresponse bias, Ph.D. thesis, University of Michigan, U.S.A.
Zajonc, T. (2012). Bayesian inference for dynamic treatment regimes: Mobility, equity and efficiency in student tracking. Journal of the American Statistical Association, 107, 80-92.
- Date modified: