6. Summary and conclusion
Natalja Menold
The results of the present study show that significant deviances from the population parameter (50% males) were seen in many ESS countries and that these were associated with the contactability or cooperation of partners in heterosexual couples (support for hypothesis H1). The magnitude of these deviances was found to differ between sampling methods when partners also differed in terms of their contactability (in subsamples with children). Thus, hypothesis H2 was partially supported. In subsamples with children, PRS was associated with the best data quality since the lowest deviances from the population parameter were found with this sampling method. However, the results for subsamples with retired partners show that highly pronounced deviances are also possible in the case of PRS.
The results for subsamples with children are in line with the explanation that interviewer behaviour related to substitutions is involved, since as expected deviations from the population parameter varied with the degree of interviewer freedom in influencing sample realization. Comparable results were reported by Sodeur (1997) and Kohler (2007). It is less plausible to explain interviewing males less than 50% of the time by differential refusal since in such cases the proportion of males is expected to be higher than 50%. Next, differential refusal is not expected to vary between different sampling methods. For retirees, interviewing males more than 50% of the time was found in several countries, but only in single rounds. This low stability of deviations from a 50% gender ratio can also be associated with interviewer impact instead of differential refusal, since the latter would be quite stable in a country over the period of time considered in the analysis. However, since the present study did not apply an experimental design it is important to address differential refusal and substitutions through further research in order to allow for better differentiation as well as for causal references.
Even though the deviations from a 50% population parameter varied in some countries across the rounds, overall their magnitude did not significantly change during the passage of time despite improved data collection procedures in the ESS (cf. Koch et al. 2009). Furthermore, deviations from the population parameter did not depend on the data collector nor were they country-specific.
The results also imply that interviewer payment and control procedures may reduce substitutions. However, it should be noted that only limited consideration of payment and control procedures was possible due to either their low variation in the data or limited information available in the survey documentation.
It should also be considered that the results presented here are based on specific subsamples and they cannot be used to generalise about the entire sample of the ESS. However, the “absence of a bias in the subsamples does not guarantee the absence of bias for the entire sample” (Kohler 2007, page 55). In addition, an analysis focused on special groups can often be of interest (e.g., what are the opinions of parents with children or of employed people).
The results of the present study imply that PRS is associated with higher sample quality, meaning a lower non-representation bias in cross-cultural surveys than with other sampling methods. This is shown more clearly by the current study than by previous studies. Analyses using ALLBUS data by Sodeur (2007) and Hoffmeyer-Zlotnik (2006) only compared several rounds of one survey in a single country (Germany); in the analysis by Kohler (2007) a sampling method effect was confounded with the survey effect (see section 1). This has been avoided in the analysis presented here.
In conclusion, significant deviations from the population parameter, which appear to be associated with substitutions by interviewers, were observed in many countries of the ESS. In order to decrease this interviewer impact it is preferable to use sampling methods, such as PRS, with which the interviewers’ degree of freedom in selecting respondents and in influencing sample quality is reduced. In addition, survey procedures that increase interviewers’ motivation to produce accurate survey data are highly relevant and should be addressed by further research as well as by survey practices.
Appendix
| Country coding: ISO 3166-1 | country | change sampling method (between rounds) | change data collector: between rounds | state pension age | |
|---|---|---|---|---|---|
| males | females | ||||
| BE | Belgium | 1-2; 2-3 | 65 | 65 | |
| BG | Bulgaria | 3-4 | 63 | 60 | |
| DE | Germany | 65 | 65 | ||
| DK | Denmark | 65 | 65 | ||
| EE | Estonia | 2-3; 3-4 | 63 | 60 | |
| ES | Spain | ARS-NRS (1-2) | 2-3 | 65 | 65 |
| FI | Finland | 65 | 65 | ||
| HU | Hungary | PRS – ARS (2-3) and back (3-4) | 2-3 | 62 | 62 |
| NO | Norway | 67 | 67 | ||
| PL | Poland | 65 | 60 | ||
| SE | Sweden | 65 | 65 | ||
| SI | Slovenia | 63 | 60 | ||
| SK | Slovakia | PRS – NRS (3-4) | 2-3; 3-4 | 62 | 59 |
| CH | Switzerland | 65 | 64 | ||
| CZ | Czech Republic | ARS-NRS (1-2) | 1-2 | 65 | 62 |
| CY | Cyprus | 3-4 | 65 | 65 | |
| GB | United Kingdom | 1-2; 3-4 | 65 | 60 | |
| GR | Greece | NRS-ARS (2-4) | 65 | 60 | |
| IE | Ireland | 3-4 | 65 | 65 | |
| IL | Israel | 67 | 64 | ||
| IT | Italy | 65 | 60 | ||
| LU | Luxembourg | 65 | 65 | ||
| NL | Netherlands | 65 | 65 | ||
| TR | Turkey | 47 | 44 | ||
| AT | Austria | 65 | 60 | ||
| FR | France | 60 | 60 | ||
| PT | Portugal | 65 | 65 | ||
| RU | Russian Federation | 60 | 55 | ||
| UA | Ukraine | 60 | 55 | ||
|
Sources: For state pension age: 1) http://www.oecd-ilibrary.org/finance-and-investment/pensions-at-a-glance-2011_pension_glance-2011-en 2) http://ec.europa.eu/employment_social/missoc/db/public/compareTables.do 3) Israel: http://www.btl.gov.il/ |
|||||
References
AAPOR (2003). Interviewer Falsification in Survey Research: Current Best Methods for Prevention, Detection and Repair of Its Effects. (http://www.aapor.org/pdfs/falsification.pdf; May 14 2009).
Arber, S. (2002). Design samples. In Researching Social Life, (Ed., N. Gilbert), Thousand Oaks: Sage, 58-84.
Billiet, J., Phillipsen, M., Fitzgerald, R. and Stoop, I. (2007). Estimation of nonresponse bias in the European Social Survey: Using information from reluctant respondents. Journal of Official Statistics, 23, 135-162.
Couper, M.P., and Groves, R.M. (1992). The role of the interviewer in survey participation. Survey Methodology, 18, 2, 263-277.
Couper, M.P., and De Leeuw, E.D. (2003). Nonresponse in cross-cultural and crossnational surveys. In Cross-Cultural Survey Methods, (Eds., J.A. Harkness, F.J.R. van de Vijver and P.Ph. Mohler), New York: John Wiley & Sons, Inc., 157-177.
De Heer, W. (1999). International response trends: Results of an international survey. Journal of Official Statistics, 15, 129-142.
De Heer, W., and Israis, A.Z. (1992). Response Trends in Europe. Paper presented to the American Statistical Association, August 1992.
De Leeuw, E., and De Heer, W. (2002). Trends in household survey nonresponse: A longitudinal and international comparison. In Survey Nonresponse, (Eds., R.M. Groves, D.A. Dillman, J.L. Eltinge and R.J.A. Little), New York: John Wiley & Sons, Inc., 41-54.
De Leeuw, E.D., and Hox, J. (1996). The effect of the interviewer on the decision to cooperate in a survey of the elderly. In International Perspectives on Nonresponse, (Ed., S. Laaksonen). Helsinki: Statistics Finland, 46-52.
Dohrenwend, B.S., and Dohrenwend, B.P. (1968). Sources of refusals in surveys. The Public Opinion Quarterly, 32(1), 74-83.
Durrant, G.B., Groves, R.M., Staetsky, L. and Steele, F. (2010). Effects of interviewer attitudes and behaviors on refusal in household surveys. Public Opinion Quarterly, 74, 1-36.
ESS1-ESS4 data from Contact forms. Bergen, European Social Survey Data Archive, Norwegian Social Science Data Services. Retrieved June 8, 2013 from: http://ess.nsd.uib.no/ess/round1/download.html.
European Social Survey Round 1-4 Data (2011). Data file edition ESS1-4e01.0_F1. Norwegian Social Science Data Services, Norway — Data Archive and distributor of ESS data.
European Social Survey (2011). ESS-4 2008 Documentation Report. Edition 4.0. Bergen, European Social Survey Data Archive, Norwegian Social Science Data Services.
European Social Survey (2011). ESS-3 2006 Documentation Report. Edition 3.3. Bergen, European Social Survey Data Archive, Norwegian Social Science Data Services.
European Social Survey (2011). ESS-2 2004 Documentation Report. Edition 3.3. Bergen, European Social Survey Data Archive, Norwegian Social Science Data Services.
European Social Survey (2011). ESS-1 2002 Documentation Report. Edition 6.2. Bergen, European Social Survey Data Archive, Norwegian Social Science Data Services.
Groves, R.M., Fowler, F.J., Couper, M.P., Lepkowski, J.M., Singer, E. and Tourangeau, R. (2004). Survey Methodology. New Jersey: Wiley.
Haeder, S., and Lynn, P. (2007). How representative can a multi-nation survey be? In Measuring Attitudes Cross-Nationally. Lessons from the European Social Survey, (Eds., R. Jowell, C. Roberts, R. Fitzgerald and E. Gillian), London et al.: Sage, 33-52.
Hoffmeyer-Zlotnik, J.H.P. (2006). Stichprobenziehung in der Umfragepraxis. Die unterschiedlichen Ergebnisse von Zufallsstichproben in face-to-face umfragen. In Stichprobenqualität in Bevölkerungsumfragen, (Eds., F. Faulbaum and Ch. Wolf). Informationszentrum Sozialwissenschaften: Bonn, 19-36.
Hox, J.J., and De Leeuw, E.D. (2002). The Influence of interviewers’ attitude and behavior on household survey nonresponse: An international comparison. In Survey Nonresponse, (Eds., R.M. Groves, D.A. Dillman, J.L. Eltinge and R.J.A. Little), New York: John Wiley & Sons, Inc., 103-118.
Johnson, T.P., O’Rourke, D., Burris, J. and Owens, L. (2002). Culture and survey nonresponse. In Survey Nonresponse, (Eds., R.M. Groves, D.A. Dillman, J.L. Eltinge and R.J.A. Little), New York: John Wiley & Sons, Inc., 55-69.
Kish, L. (1965). Survey Sampling. New York: John Wiley & Sons, Inc.
Koch, A. (1995). Gefälschte Interviews: Ergebnisse der Interviewerkontrolle beim ALLBUS 1994. ZUMA-Nachrichten, 36, 89-105.
Koch, A., Blom, A.G., Stoop, I. and Kappelhof, J. (2009). Data collection quality assurance in cross-national surveys at the example of the ESS. Methoden Daten Analysen, 3, 219-247.
Kohler, U. (2007). Surveys from inside: An assessment of unit nonresponse bias with internal criteria. Survey Research Methods, 1, 2, 55-67.
Kreuter, F., and Kohler, U. (2009). Analyzing contact sequences in call record data. Potential and limitations of sequence indicators for nonresponse adjustments in the European Social Survey. Journal of Official Statistics, 25, 203-226.
Lynn, P., Haeder, S., Gabler, S. and Laaksonen, S. (2007). Methods for achiving equivalence of samples in cross-national surveys: The European Social Survey Experience. Journal of Official Statistics, 1, 107-124.
Pickery, J., and Loosveldt, G. (2002). A multilevel multinomial analysis of interviewer effects on various components of unit non response. Quality and Quantity, 36, 427-437.
Philippens, M., and Billiet, J. (2004). Monitoring and evaluating nonresponse issues and fieldwork efforts in the European Social Survey. Paper presented at the European Conference on Quality and Methodology in Official Statistics. Mainz, Germany.
Schnauber, A., and Daschmann, G. (2008). States oder traits? Was beeinflußt die Teilnahmebereitschaft an telefonischen Interviews? Zeitschrift Für Empirische Sozialforschung, 2, 97-123.
Schnell, R., Hill, P.B. and Esser, E. (2011). Methoden der Empirischen Sozialforschung. München: R. Oldenbourg Verlag.
Smith, T.W. (2007). Survey nonresponse procedures in cross-national perspective: The 2005 ISSP non-response survey. Survey Research Methods, 1, 45-54.
Sodeur, W. (1997). Interne kriterien zur Beurteilung von Wahrscheinlichkeitsauswahlen. ZA-Information, 41, 58-82.
Sodeur, W. (2007). Entscheidungsspielräume von Interviewern bei der Wahrscheinlichskeitsauswahl. Methoden Daten Analysen, 1, 2, 107-130.
Stoop, I.A.L. (2004). Surveying nonrespondents. Field Methods, 16, 23-54.
Stoop, I.A.L. (2005). The Hunt for the Last Respondent. The Hague: Social and Cultural Planning Office.
Symons, K., Matsuo, H., Beullens, K. and Billiet, J. (2008). Response Based Quality Assessment in the ESS — Round 3: An Update for 19 countries. London: Centre forComparative Social Surveys, City University.
Vehovar, V. (1999). Field substitution and unit nonresponse. Journal of Oficial Statistics, 2, 335-350.
Williams, B., Irvine, L., McGinnis, A.R., McMurdo, M.E.T. and Crombie, I.K. (2007). When “no” might not quite mean “no”; the importance of informed and meaningful non-consent: results from a survey of individuals refusing participation in a health-related research project. BMC Health Services Research, 7, 59.
- Date modified: