Coverage and frames

View the most recent version.

Archived Content

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please "contact us" to request a format other than those available.

Archived

This page has been archived on the Web.

Scope and purpose
Principles
Guidelines
Quality indicators
References

Scope and purpose

The target population is the set of units about which information is wanted and estimates are required. Practical considerations can dictate that a survey population be defined which excludes some units in the target population or which is comprised of differently defined units through which the target population can be accessed.

A frame is any list, material or device that delimits, identifies, and allows access to the elements of the survey population. Frames are generally of two types: area frames and list frames. A list frame is a list of units in the survey population. Area frames are usually made up of a hierarchy of geographical units which in turn contain units in the survey population; that is, the frame units at one level can be subdivided to form the units at the next level. All of the elements included in the frame constitute the frame population. Frames are often much more than a simple list of units or a map with geographic units delineated. A frame usually includes other information (e.g., identification, contact, classification, address, size, maps in case of geographical units) to be used in carrying out the survey.

Coverage is the completeness of the information for the target population that would be derived if all of the frame units were to be surveyed. Coverage errors are discrepancies in statistics for the target population versus those for the frame population. These errors are a function of both the frame undercoverage (or overcoverage) of the target population and of coverage errors occurring during survey operations resulting in differences in the survey estimate for those actually covered from those for which an estimate was required.  Coverage errors can have both spatial and time dimensions.

Principles

The survey population should be reasonably consistent with the target population in order for the survey results to be relevant.

In turn, the survey frame should conform to the survey population. Frame coverage errors (such as missing in-scope units, included out-of-scope units, misclassified units and duplicates) can complicate the survey process resulting in cost increases, loss of timeliness  and also will diminish the accuracy (from bias and variance points of view) of the estimates.

Frame data should be up-to-date and accurate because of their use in stratification, sample selection, collection, follow-up, data processing, imputation, estimation, record linkage, quality assessment and analysis.  Erroneous frame data are likely to bias or diminish the reliability of the survey estimates and to increase data collection costs.

Survey designs and operations should implement procedures to minimize coverage error and its impact.

Guidelines

Design

• Test possible frames at the planning stage of a survey for their suitability and quality. Assess the coverage of the frame and of the target collection units.

• When no single frame can provide the required coverage of the target population, use a multiple frame methodology. A multiple frame is a combination of two or more frames such as a list and area frame or two or more list frames.  Generally avoid using multiple frames unless no single existing frame is adequate.  When several frames exist, some of which are incomplete but less expensive to use and others more complete but prohibitively expensive, consider the use of multiple frames.

• Consider also the use of Random Digit Dialling (RDD) for some telephone surveys, by itself or in combination with other area or list frames.

• Sometimes no cost effective frame exists for the population of units of interest. In such situations consider using multi-stage or indirect sampling methods.

• At Statistics Canada, several lists are maintained for use as frames by its surveys. The Business Register is available for surveys of businesses and institutions. For agricultural surveys, the Farm Register is the usual frame. For household surveys, the Address Register, the Labour Force Survey frame (which is an area frame), and the Census of Population geographic units are options to consider. In situations where one of these recognized frames is not the best choice for addressing a survey's target population, other frames (e.g. lists of immigrants or databases of importers or exporters) should be considered.

• Ensure that the frame is as up-to-date as possible relative to the reference period for the survey.

• Retain and store information about sampling, rotation and data collection so that coordination between surveys can be achieved and respondent relations and response burden can be better managed.  For example, record how often each unit is selected by each survey that is using the frame.

• For statistical activities from administrative sources or for derived statistical activities, where coverage changes may be outside the control of the immediate manager, determine and monitor coverage, and negotiate required changes with the source manager.

• Make adjustments to the data or use supplementary data from other sources to offset coverage error of the frame.

• Where possible, use the same frame for surveys with the same target population to improve coherence, avoid inconsistencies, and facilitate combining estimates from the surveys to reduce costs of frame maintenance and evaluation.

• To create geographic frame units at Statistics Canada, use the Generalized Area Delineation System (GArDS), a partially generic auto-spatial delineation and verification system for creating the non-overlapping contiguous geographic frame units.

• When multiple frames exist, they can be used to assess the completeness of one of the frames.

• Implement survey procedures to detect and correct coverage errors from the frame. Provide feedback to update and maintain the frame.

• Implement training and procedures for data collection and data processing staff aimed at minimizing coverage error (e.g. procedures to ensure accurate confirmation of lists of dwellings for sampled area frame units).

• Design survey questionnaires and related materials so as to minimize coverage errors committed by respondents (erroneous listing on a questionnaire of persons in a dwelling, omission of in-scope locations in an establishment survey).

Maintenance

• To improve and/or maintain the level of quality of the frame, incorporate procedures to eliminate duplication and to update for births, deaths, out-of-scope units and changes in characteristics.

  • Incorporate frame updates in the timeliest manner possible.

• Minimize frame errors through effective training of staff, an emphasis on the importance of coverage, and the implementation of quality assurance procedures for frame-related activities.

• For area frames, implement map checks to ensure clear and non-overlapping delineation of the geographic areas used in the sampling design (e.g., through field checks or the use of other map sources). When appropriate, use the information from the address register to verify field listing of residential addresses for undercoverage or overcoverage.

• Review and improve the identification of the target units missed or wrongly coded and put in place procedures to minimize this problem.

• Put in place procedures to detect and minimize errors of omission and misclassification that can result in undercoverage, and to detect and correct errors of erroneous inclusion and duplication resulting in overcoverage. 

Documentation

• Include definitions of the target and survey populations, any differences between the target population and the survey population, as well as the description of the frame and its coverage errors in the survey documentation.

• Report known gaps between key user needs and survey coverage

Quality indicators

Main quality elements:  accuracy, relevance

Coverage errors arise from both undercoverage and overcoverage. In addition to target population versus survey population differences, the former occur when units are erroneously omitted from the frame file while the latter occur when units are incorrectly included on the frame file (e.g. dead units). Classification errors – in industry for example – result in coverage errors on the frame; undercoverage in the "correct" classification and overcoverage in the "incorrect" classification.  Classification information can be used in stratification, sample selection, data processing, imputation, estimation, record linkage, and quality assessment and analysis.  Contact information can be used in data collection and follow-up. Frame imperfections such as coverage errors and outdated characteristics are likely to bias or diminish the reliability of the survey estimates and to increase data collection costs. 

Under and overcoverage can undermine both the relevance and the accuracy of survey results.

• Monitor the frame quality by periodically assessing its coverage and the quality of the information on the characteristics of the units. Many techniques exist for this purpose:

• matching the frame or a sample of the frame with comparable alternative sources, often provided by administrative records, for the survey population or subsets of it;

• analyzing survey returns for duplicates, deaths, out-of-scope units, and changes in characteristics;

• using specific questions on the questionnaire to aid in monitoring coverage and classification information; verifying with local authorities (e.g., regional offices, field survey staff, the survey units themselves);

• verifying the frame or subsets of it in the field (which could include verification of out-of-scope units);

• comparing the frame with a sample of units from a corresponding area frame;

• updating the frame to determine changes to it;

• checking the consistency of counts with other sources or with data from specially designed replicates;

• using evaluative information obtained from other surveys with the same frame (Lessler and Kalsbeek, 1992).

• Monitor the frame between the time of sample selection and the survey reference period.

• Define and compare the target population and the survey population.

• Measure coverage error in censuses via Post-enumeration (Hogan, 2003) or Reverse Record Check surveys (Statistics Canada, 2004) and associated studies of overcoverage. For censuses of population and housing, coverage error can also be assessed via comparison of census counts to demographic estimates. Provide estimates not only of net error but also of components of error.

• Provide estimates of coverage error or slippage rates both for current users of survey estimates and also for designers of future iterations of the same survey.

References

Archer, D. 1995. "Maintenance of business registers." Business Survey Methods. B.G. Cox et al (eds.) New York. Wiley-Interscience, p. 85-100.

Hartley, H.O. 1962. "Multiple frame surveys." Proceedings of the Social Statistics Section. American Statistical Association. p. 203-206.

Hogan, H. 2003. "The Accuracy and Coverage Evaluation: Theory and Design." Survey Methodology, Vol. 29, no. 2. p. 129-138.

Kott, P.S. and F.A. Vogel. 1995. "Multiple-frame business surveys." Business Survey Methods, B.G. Cox et al (eds.) New York. Wiley-Interscience, p. 185-203.

Laniel, N. and H. Finlay. 1991. "Data quality concerns with sub-annual business survey frames." Proceedings of the Section on Survey Research Methods. American Statistical Association. p. 202-207.

Lessler, J.T. and W.D. Kalsbeek. 1992. Nonsampling Errors in Surveys. New York. Wiley-Interscience. 432 p.

Massey, J.T. 1988. "An overview of telephone coverage." Telephone Survey Methodology. R.M. Groves et al (eds.) New York. Wiley-Interscience. p. 3-8.

Statistics Canada. 2003. Survey Methods and Practices. Statistics Canada Catalogue no. 12-587E-XPE. Ottawa, Ontario. 396 p.

Statistics Canada. 2004. 2001 Census Coverage Technical Report. Statistics Canada Catalogue no. 92-394-XIE. Ottawa, Ontario. 85 p.

Statistics Canada. 2008. Methodology of the Canadian Labour Force Survey. Statistics Canada Catalogue no. 71-526-X. 116 p.

Swain, L., J.D. Drew, B. Lafrance and K. Lance. 1992. "The creation of a residential address register for coverage improvement in the 1991 Canadian Census." Survey Methodology. Vol. 18, no. 1. p. 127-141.