Section 3
Coverage errors

View the most recent version.

Archived Content

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please "contact us" to request a format other than those available.

Archived

This page has been archived on the Web.

Undercoverage and overcoverage: slippage rates
Adjustment at the population and household levels

The target population was defined in the design of the survey. It is useful to go over this definition, since a good understanding of the target population is necessary in order to properly interpret the survey data. It is important to note that the Survey of Household Spending (SHS) uses the sampling frame of the Labour Force Survey (LFS).

Target population

We did not collect data from persons temporarily living away from their families (for example, students at university) because the information would be obtained from their families if selected in the sample.

Coverage errors result from inadequate representation of the target population based on the units in the sampling frame. Some units of the target population may be omitted from the sampling frame, in which case there is undercoverage. Other units that are not in the target population may be included by error, or some units may be included more than once. These units are responsible for overcoverage.

3.1 Undercoverage and overcoverage: slippage rates

In the SHS, the sample is selected using a list of dwellings in each selected cluster. Factors contributing to undercoverage are: the omission of dwellings in the creation of the list, new dwellings that are added between the creation of the list and the interviewer's visit (mainly in developing areas), and the erroneous classification of vacant dwellings. The inclusion of dwellings that are not within the boundaries of the cluster is a source of overcoverage. Similarly, errors can occur during data collection, due to improper identification of persons as members of the selected household. These errors also contribute to undercoverage or overcoverage.

Also, as described in Section 2.4, reweighting methods are implemented to take account of nonresponse. However, when these adjustments are made, it is impossible to correct the survey weights to ensure that all subgroups within the population are well represented. 

A good representation of the target population is essential to the production of realistic expenditure estimates. The sample must adequately represent the individuals in the target population as well as the distribution of households according to their size.

There is generally a net undercoverage of the number of persons and the number of households in the SHS. This undercoverage is corrected by an adjustment of weights using auxiliary or reference data based on post-censal demographic estimates. The slippage rate (see Appendix A) is a measure of the percentage of difference between the estimates from these auxiliary data and the survey estimates calculated using weights not adjusted with these data². Slippage therefore represents the combined effect of undercoverage and unbalance in certain subgroups of the population created by survey nonresponse that could not be corrected at the reweighting stage.

Slippage rates by age group at the national and provincial level are shown in Table 3.1, while slippage rates by household size, used in adjusting weights, are shown in Table 3.2. A positive rate indicates overcoverage of the number of persons or households in the survey.

Table 3.1 Slippage rates for provinces and territories by age group

For the 2005 SHS, the net undercoverage rate for the population was 11.3%, excluding the territories.  An analysis of Table 3.1 with respect to age groups reveals that at the national level (excluding the territories), the slippage rates for children (aged 0 to 6 and 7 to 17) are very different from those for the other age groups. The net undercoverage rate for all children combined is 4.2%, while it is 13.2% for adults (data not shown).  The slippage rates for those aged 55 and over are also lower than for other adults. The highest rates at the national level are for individuals in the 18 to 24 and 25 to 34 age groups.

We also observe net undercoverage for all provinces, with the rates varying from 6.1% to 17.3%. Quebec has the lowest net undercoverage rate (6.1%). Note, however, that a low overall undercoverage rate does not guarantee better coverage for all subgroups in the population. For example, the general slippage rate observed in Quebec conceals one of the worst cases of overcoverage at the provincial level for the 0 to 6 age group (4.8%) and one of the worst cases of undercoverage for the 65 and over age group (9.0%). The highest net undercoverage rate is in Saskatchewan, where it stands at 17.3%.

When we analyse the cross-tabulation of provinces and age groups, we observe that the highest net undercoverage rate is for the 18 to 24 age group in Saskatchewan (40.5%). Another point worth noting is that the pattern of slippage rate variation differs substantially for age groups from one province to the next. However, the worst undercoverage rates are generally observed in the 18 to 24 and 25 to 34 age groups. The lowest rates are for the 0 to 6 and 7 to 17 age groups, as was seen at the national level.

As mentioned previously, the SHS uses the LFS sampling frame. Over the same period, the national LFS undercoverage rate was 9.4% (reference [5]). This is lower than the 13.4% SHS rate for those aged 15 and over.

Table 3.2 Slippage rates for provinces and territories by household size

As the table shows, there is 8.2% undercoverage of the number of households nationally. This undercoverage is slightly lower than the undercoverage of 11.3% observed for the number of individuals. Both nationally and provincially, undercoverage is generally observed for all sizes of household. Nationally, the undercoverage rate for one-person households (5.7%) is comparable to the corresponding rate for two-person households (5.8%) but is approximately half the undercoverage rate for households of three or more persons (12.1%).

Provincially, there is also a sizable variation in slippage rates, with rates varying from -5.1% in Quebec to -12.0% in New Brunswick. These rates are generally consistent with slippage rates for individuals as seen in Table 3.1.

While the undercoverage of one-person households is relatively low at the national level, it is the largest component of household undercoverage for six of the thirteen provinces and territories. Overcoverage of one-person households in Saskatchewan (4.6%) and Alberta (9.7%) tends to offset in part the contributions of the other provinces and territories to national undercoverage.

3.2 Adjustment at the population and household levels

To correct the problem of the sample's representativeness, shown in Table 3.1, and to reduce the resulting bias, the survey data are adjusted during weighting using demographic estimates for the age groups defined in this table, for each province. For more details on the adjustment methodology, see references [1] and [6]. This adjustment reduces the bias but does not eliminate it entirely if the characteristics of the individuals omitted from the survey differ from those of individuals included for a given age group in a province.

It should also be noted that the effectiveness of the adjustment based on demographic estimates depends largely on the quality of those estimates and their accuracy in representing the target population of the survey.  The demographic estimates are not error-free. They are post-censal estimates based on the population counts from the 2001 Census adjusted for net undercoverage, and they take into account recent statistics on migration, births, deaths, etc. These demographic estimates are adjusted to account for certain exclusions specific to household surveys, such as persons living in institutions. Conceptually, they differ slightly from the SHS target population in that they include persons living in non-institutional collective dwellings, such as members of groups living communally and individuals permanently residing in hotels or rooming houses. However, this difference is considered negligible, since such individuals represent less than 0.4% of the Canadian population.

To remedy the problem of the representativeness of the sample in terms of the number of households by size as illustrated in Table 3.2, the survey data are adjusted using supplementary data. By adjusting the SHS weights to reflect post-censal estimates of the number of households by size, the goal is to compensate for the bias resulting from inadequate representation of households.  However, the bias will not necessarily be eliminated if characteristics of households not interviewed (i.e., omitted or non-responding households) differ from those of responding households for a given household size. As in the case of demographic estimates of population, the effectiveness of the adjustment will depend on the quality of the supplemental data on the number of households. 

In addition to demographic estimates of age groups by province, two other sets of supplementary data are used during weighting to adjust survey data and thereby improve their representativeness.  The first set of data is used to control for the number of children and adults in certain major cities. For the second set, counts for major categories of income from wages and salaries are used when adjusting weights to ensure a degree of consistency between the income distributions from the SHS and those from outside sources.

---

Notes

1. As to the proportion of households, the coverage in Yukon, the Northwest Territories and Nunavut is respectively 92%, 93% and 73% of households.
2. The subweight, which is the survey weight adjusted for nonresponse, is used (see Appendix A).