2. Theoretical and empirical background
Natalja Menold
Substitutions may arise during tasks that interviewers conduct prior to the interview. Interviewers (1) build a sampling frame, for example by generating lists of addresses in surveys; then they (2) elicit cooperation with selected units (addresses, dwellings, households) and they also select individuals for the interview from these units. Finally interviewers (3) elicit cooperation from sampled individuals (Groves et al. 2004). In the case of different sampling methods interviewers perform different tasks as described below (figure 2.1).
The first sampling method refers to individual person register samples (denoted below as PRS). Official population registers of individuals are used as sampling frames for PRS. The selection of individuals is conducted prior to the field work stage, thus reducing interviewers’ tasks to simply obtaining cooperation from sampled individuals (figure 2.1). In the case of PRS interviewers may influence non-response (e.g., Couper and Groves 1992; de Leeuw and Hox 1996; Durrant, Groves, Staetsky and Steele 2010), but in a theoretical sense they have no influence on the sampling frame or on the selection of sample elements. This level of interviewer impact on non-representation in the case of PRS is shown by the arrow in figure 2.1.
However, as shown for example by Groves et al. (2004) selected elements (individuals in the case of PRS) may differ in terms of the probability of being contacted by an interviewer (contactability) and the probability of obtaining survey participation when a contact is given (cooperation). For example, it has been found that people living in urban areas or individuals who are young, single, without children, better educated and socially active are more difficult to contact (Stoop 2004). In contrast, older respondents, women, less-educated people and socially isolated individuals refuse cooperation more often than others (Dohrenwend and Dohrenwend 1968; Stoop 2004; Williams, Irvine, McGinnis, McMurdo and Crombie 2007). If difficulties arise when trying to contact and obtain cooperation from target persons then substitutions may occur. For example, Koch (1995) reported the number of substitutions in a survey in which PRS was used.
The next sampling methods discussed are address/household register samples (ARS). In the case of ARS lists of households or addresses are employed as sampling frames. Households or addresses are selected by survey offices prior to the field work stage. In this case interviewers perform tasks two and three (see above): they contact selected units and select individuals for the interview if more than one eligible individual is living at one unit. Interviewers may deliberately deviate from the random selection rules and in this way they can have a negative impact on the selected sample (figure 2.1). Since interviewers have more freedom in selecting sampled individuals with ARS than with PRS, it is assumed that interviewer impact associated with substitutions is higher for ARS than for PRS (figure 2.2). Moreover, in the case of ARS the result of sample selection is not known beforehand and is therefore harder to control than in the case of PRS.
Non-register samples (NRS), in which neither lists of individuals nor lists of addresses/households are available as sampling frames, are described as the third sampling method. These include Random Route Samples (e.g., Arber 2002; ESS Sampling Plans), and Address Listing and Sampling (ALS). In the case of NRS interviewers themselves generate a sampling frame by listing or collecting addresses within a randomly selected geographical area. Interviewers have to strictly follow instructions concerning the procedures for collection of addresses. Interviewers perform this task in addition to selecting individuals at one address, as described for ARS, and to contacting and obtaining cooperation, as just described for both PRS and ARS (figure 2.1). With NRS interviewers can influence not only sample selection but also the sampling frame. An interviewer can deviate from instructions and chose only addresses where he or she expects to contact the target person and obtain cooperation. Substitutions are particularly likely to occur when using a procedure (Random Route) in which the interviewer conducts the interview at an address which they choose in an area following prescriptions regarding collecting of addresses and the direction of the route through the area. Another type of NRS is more restricted since the interviewer lists the addresses in a geographical area, but the actual selection is conducted by a coordination team (Address Listing and Sampling, ALS). The selected addresses are subsequently assigned to a different interviewer who then conducts the interviews. The degree of interviewer freedom in the case of ALS appears to be similar to that of ARS. However, the instructions for listing or collecting addresses can be ambiguous in the case of both types of NRS (Schnell, Hill and Esser 2011). Therefore, interviewers have more freedom to substitute with NRS than with ARS (figure 2.2).
Deviations in obtained net samples, which are associated with deviations from the rules of random sample selection (e.g., substitutions), can be empirically analysed using a method developed by Sodeur (1997). This method works by defining a subpopulation with one fixed and known parameter, then the statistics representing this parameter are observed in a subsample that is defined in a corresponding manner. The more the observed statistic deviates from the population parameter, the stronger the error from non-observation is. This article considers the gender ratio of heterosexual couples, which is known to be a 50/50 population parameter. Within the limits of random fluctuation any sample from the population of heterosexual couples should produce a proportion of males of around 50%. Significant deviations from this level of 50% indicate deviations from sample selection standards, such as through substitutions (see section 4.2 for details).
Using this method Sodeur (2007) and Hoffmeyer-Zlotnik (2006) found that deviations from the true parameter of a 50/50 gender ratio vary between different rounds of the German General Social Survey (ALLBUS), which also involved different sampling methods. The authors found that males who are difficult to contact are interviewed less often than females (since the males are the breadwinners in households with children). Aside from contactability, differences in cooperation between the partners can play a role (Hoffmeyer-Zlotnik 2006). If partners are retired they have comparable levels of contactability but they may differ in terms of cooperation. The retired man, now at home, feels responsible for providing information to an interviewer about the household (as its “head”). The woman can refuse to participate since the man likes to cooperate. An interviewer who contacts such households may interview men instead of women in order to avoid refusals (Hoffmeyer-Zlotnik 2006).
Kohler (2007) found larger deviations from the parameter of a 50/50 gender ratio in NRS samples as compared to other sampling methods in six cross-cultural surveys (Eurobarometer 62.1, European Quality of Life Survey EQLS’03, ESS 2002, ESS 2004, European Value Study 1999, International Social Survey Program, ISSP 2002). Unfortunately, the sampling method effect obtained by Kohler (2007) was survey-specific. The most poorly designed samples—area samples with a NRS—were used predominantly in one survey (EQLS). The differences that Kohler found between a random route and other sampling methods could thus be due to differences between EQLS and other surveys. Other researchers (Hoffmeyer-Zlotnik 2006; Souder 1997) addressed the effect of sampling methods on interviewer impact associated with substitutions while considering only a single German national survey; the results of this research are not applicable to cross-cultural contexts. Therefore, it is important to address the question regarding the relationship between sampling methods and interviewer impact associated with substitutions in cross-cultural surveys. Additionally, it is important to consider other explanatory factors which can affect substitutions. Substitutions made by interviewers can be affected not only by sampling methods but also by field procedures related to interviewer motivation to produce accurate survey data. Therefore, substitutions may vary based on the data collector (Hoffmeyer-Zlotnik 2006; Sodeur 1997; 2007) or controls used in a survey (Kohler 2007). Controls imply that a sample element is re-contacted to confirm the outcome produced by an interviewer. In addition to controls, methods of providing payment to interviewers can impact their performance. If interviewers are paid per completed interview they bear the risk of high costs due to long distances between selected addresses, numerous contact attempts or long interview times (Sodeur 2007). Consequently, a change of data collector, controls and payment should be considered when analysing the interviewer impact associated with substitutions. Apart from these factors it is interesting to see how such interviewer impact varies across time. For example, considering time in a cross-cultural context helps to indicate whether this interviewer impact is country-specific. A country-specific interviewer impact should be stable in a country across different survey rounds, even if the sampling method is changed.
- Date modified: