4. Methods

Natalja Menold

Previous | Next

4.1 Data

To isolate any effects due to the sampling method from other survey-specific effects, one can use data from a multi-country survey in which the various countries applied different sampling methods. Many rounds of a survey should be available in order to be able to consider the time effect. Therefore, data from rounds one to four of the ESS were used (European Social Survey Round 1-4 Data 2011). The ESS was conducted with between 20 and more than 30 countries, which differ in terms of their sampling methods. In addition, the ESS sets high standards for survey organisations, such as strict random sampling and extended contact procedures, or regarding field control procedures (Koch, Blom, Stoop and Kappelhof 2009; Philippens and Billiet 2004). The effectiveness of the standards used in the ESS has been demonstrated by Kohler (2007), who showed that round 1 of the ESS had the fewest deviations from the 50/50 gender ratio compared to other surveys. In addition, the ESS has consistently improved data collection methods (Koch et al. 2009). Furthermore, the ESS provides detailed documentation of sampling procedures as well as data collection (cf. ESS Documentation Reports), which allows for operationalization of variables of interest.

4.2 Method for evaluating interviewer impact

The method developed by Sodeur (1997) was used for the analysis. This method helps to evaluate the net sample quality in probability samples. The quality of the random sample selection has often been examined by means of other statistics available in a country (external criteria). However, these external statistics are often unknown, leading Sodeur to suggest the use of internal criteria - that is to say use of information from the net sample only. Sodeur (1997) describes the method as consisting of the following steps: (1) separating a subsample from the entire sample to focus on respondents as representatives of heterosexual couples: the partners should live together within one household and both partners should belong to the target population of the survey; (2) defining units which should be dropped from the subsample: singles, partners not living together within a household and households with other relatives who belong to the target population. Then, step three entails (3) defining a survey statistic — e.g., the percentage of males — as the dependent variable which should be compared with the population parameter.

An analysis to determine the causes of deviations from the population parameter — for example, interviewer behaviour — requires additional specifications in steps 1 and 2 to ensure that interviewer behaviour (conceptually) varies with the contactability or cooperation of target persons. Such specifications have been made in this article in terms of definitions of different type of households (see hypothesis H1), whose selection is described in section 4.3.

The true 50/50 gender ratio in heterosexual couples is not related to any other gender ratios, such as that of the total population of residents in a country. Therefore, as Kohler (2007) argues, this gender ratio cannot be affected by any sort of measurement errors and it is unaffected by the household size since the analysis is restricted to two persons in the household and both persons belong to the target population.

Sodeur's method has advantages over other methods since no additional external information or data are required. However, Sodeur's method requires that the characteristics defined for selecting subsamples are known not only regarding the respondents but also regarding their partners (e.g., gender of the partner). In addition, there should not be systematic gender differences in terms of refusal behaviour (differential refusal), which may occur even if interviewers work honestly. In practice, females have been found to be more reluctant than males (Pickery and Loosveldt 2002; Schnauber and Daschmann 2008; Stoop 2004; Williams et al. 2007). That also seems to be the case in the ESS, in which females were found to refuse more often than males. The author's own analysis of ESS1-ESS4 data from contact forms shows that 30.3% males and 37.9% females refused cooperation in the ESS1 (in some countries no data regarding the gender variable was provided; therefore the missing data was 32.4%). In the ESS2 there were 30.8% males and 37.9% females who refused cooperation (31.3% data missing); in the ESS3 33.8% males and 39.0% females (27.2% of data missing) refused cooperation and in the ESS4 there were 38.4% males and 45.8% females who refused (with a reduced 15.8% of data missing). Therefore, males being present in a subsample of ESS data less than 50% of the time can be plausibly explained by substitutions, while a frequency of males higher than 50% can be alternatively explained by differential refusal. However, if the percentage of males varied with a sampling method — as expected in hypothesis H2 — it would be hard to explain such a result only by differential refusal, which seems to be a quite stable feature.

4.3 Procedure

The following section provides a description of the procedures used for testing hypotheses H1 to H3. First, separation of subsamples from the entire ESS sample is described. Deviations $d$ from the true 50/50 gender ratio in a subsample represent the dependent variable for all subsequent analyses. The values of $d$ are compared between different households to test hypothesis H1. Second, operationalization of the "sampling method" variable (to test H2) is described. Third, hypothesis H3 is related to the variables time, change of data collector, payment and interviewer controls, whose operationalization is described in the last section. H2 and H3 were tested with the help of a Multivariate Analysis of Covariance (MANCOVA) with subsequent individual Analyses of Covariance (ANCOVA) in which the sampling method was used as the independent variable and the ESS round, change of data collector, payment bonus and interviewer controls were used as covariates.

Separation of subsamples

The ESS target population "contains in each country persons 15 years or older who are resident within private households, regardless of nationality and citizenship, language or legal status" (e.g., ESS-1 2002 Documentation Report, page 2). Respondents ( $n=$ 88,375) who live together with a partner of the opposite sex who is 15 years or older were selected from the total ESS1-ESS4 sample ( $n=$ 184,988). This reduced the data base of the analysis to about half of the entire sample. However, such a selection is required to ensure the expected percentage of males of 50%.

Three household types were distinguished among the selected subsample: couples with children aged between 0-6 ($<7;n=$18,791), couples with children between 7-14 ($n=$53,651), and couples in which both partners are of retirement age (retirees, $n=$15,933). To determine retirement age the current state pension age in each country was used (see appendix). The first two groups with children were formed since it was assumed that differences in contactability between partners are particularly high in these households. For the third group it was assumed that gender differences in contactability are fairly modest, while men and women might differ in terms of cooperation.

The fact that men are breadwinners within the two subsamples containing households with children is supported by the author's own analysis using data from the ESS. Upon looking at respondents' activities within the last seven days in households with children younger than seven, it was shown that 58% of males and 42% of females were in paid work. In terms of respondents' partners, 64% of males and 36% of females were in paid work. Similar results were found for respondents in households with children aged 7 to 14 (for respondents 54% males and 46% females were in paid work and in terms of partners there were 60.5% males and 39.5% females). For households with retired partners it was found that 80.6% of respondents were retired, 11.5% did housework and 1.3% was sick or disabled on a long term basis. With respect to respondents' partners, 84.4% were retired, 17% did housework and 2.1% were sick or disabled on a long-term basis.

Categorisation of sampling methods

Documentation Reports for each ESS round (European Social Survey (2011): ESS 1-4 Documentation Reports) were used to classify the sampling methods. Table 4.1 summarises the main characteristics of the sampling methods used in the ESS. Table 4.2 shows which sampling methods were used in each country in each of the rounds. For more details on selection procedures in the ESS see the Documentation Reports or Lynn et al. (2007).

For ARS it is important to see how multi-dwelling units at one address are dealt with since in this case interviewers also manage the situation. The survey documentation described this for only a few countries (Ireland, Israel, the Netherlands, and the United Kingdom). In Ireland, for example, interviewers listed the households and selected one of them using the Kish Grid (Kish 1965).

In Austria a NRS method was applied to only 50% of the sample, while the other 50% was selected based on an ARS. Since using NRS can lead to more substitutions compared to using only ARS it can be expected that the results in Austria are more similar to the results in countries with NRS than with ARS. Therefore, the author assigned Austria to NRS.

Table 4.1
Sampling methods in the countries of the EES (rounds 1-4)
Table summary
This table displays the results of Sampling methods in the countries of the EES (rounds 1-4) individual person register sample, address/household register sample and non-register sample (appearing as column headers).

	individual person register sample	address/household register sample	non-register sample
sampling frame	reliable lists of residents	reliable lists of addresses/households	regional areas (no lists of residents, addresses or households)
stage 1: Selection of PSUs
definition of a unit	regional clusters, areas, municipalities	electoral sections, postal code areas	regional clusters, areas, municipalities
process of selection	systematic random sampling	systematic random sampling	systematic random sampling
result	community, municipality	electoral section, postal code section	geographical areas, municipalities
stage 2: Selection of households
definition of a unit	not applicable	a household, an address	a household/dwelling unit
process of selection	not applicable	simple or systematic random sampling	random route/ALS simple random sampling
result	not applicable	addresses of households	a household/address/dwelling unit
stage 3: Selection of individuals
definition of the unit	target person	target person	target person
process of selection	simple or systematic random sampling	random selection by interviewer by Kish Grid or last birth day method	random selection by interviewer by Kish Grid or last birth day method
result	name and address of sampled individuals	sampled individuals	sampled individuals

Table 4.2
Classification of ESS countries with respect to sampling methods
Table summary
This table displays the results of Classification of ESS countries with respect to sampling methods. The information is grouped by ESS round (appearing as row headers), individual person register sample (PRS), address/household register sample (ARS) and non-register sample (NRS) (appearing as column headers).

ESS round	individual person register sample (PRS)	address/household register sample (ARS)	non-register sample (NRS)
ESS 1	BE, DE, HU, NO, PL, SI, DK, FI, SE	Address: IE, IT, NL, GB, CH Household: CZ, LU, ES	FR, GR, PT, AT
ESS 2	BE, DE, HU, NO, PL, SI, DK, FI, SE, ES, EE, IS, SK	Address: IE, NL, GB, CH Household: LU, TR	FR, GR, PT, AT, CZ, UA
ESS 3	BE, DE, NO, PL, SI, DK, FI, SE, ES, EE, SK	Address: IE, NL, GB, CH, LV Household: CY, BG, HU	FR, PT, AT, UA, RU, RO
ESS 4	BE, DE, HU, NO, PL, SI, DK, FI, SE, ES, EE	Address: IE, NL, GB, CH, IL, LV Household: CZ, CY, LT, GR, KRO, TR, BG	FR, PT, SK, UA, RU, RO
Note: Romania is not included in the ESS integrated data file; no information on sampling method has been provided for Italy in Documentation Reports ESS2-ESS4. Countries are labelled according to ISO 3166-1, see the appendix.

The kind of NRS method used in a country has rarely been described in the documentation. In the ESS1 it is evident that an ALS was used only in Greece. Usage of an ALS is described for the Czech Republic and Slovakia in later rounds. In the ESS4 Ukraine, Russia and Portugal report a procedure comparable to the ALS. However, in these countries interviewers (and not the offices) selected a fixed number of units from the lists generated by other interviewers.

Explanatory variables

Information related to the particular ESS round was used as a variable to control for the time effect. The Documentation Reports were used to obtain information related to other explanatory variables; change of data collector, payment and interviewer controls. Whether a country changed the data collector between rounds is shown in the appendix. Concerning payment, it was found that the ESS mainly employed a method involving payment per completed interview. An hourly rate of payment was used only in a few countries that also used PRS (in ESS1-2 in Norway and Sweden, as well as in ESS3-4 in Norway and Finland). Therefore, there was only a small variation in payment methods, and a corresponding data analysis was not possible. However, payment of bonuses varied across countries and rounds. Therefore, this information was used to generate a dichotomous control variable (bonus payment: yes/no).

Two variables are used to describe control procedures: the number of eligible sample elements selected for controls divided by the number of eligible sample elements (ratio selected), and the number of confirmed outcomes divided by the number of sample elements selected for controls (ratio confirmed). The first variable describes the number of controls in a country, while the second describes the effectiveness of these controls. The "ratio selected" varies between 10% for PRS, 13% for NRS and 16% for ARS. The "ratio confirmed" is somewhat higher for NRS ($M=$75.21, $SD=$24.81) than for the other two sampling methods (PRS: $M=$61.89, $SD=$31.95; ARS: $M=$66.49; $SD=$32.56).

Previous | Next