Sample design

View the most recent version.

Archived Content

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please "contact us" to request a format other than those available.

Archived

This page has been archived on the Web.

Scope and purpose
Principles
Guidelines
Quality indicators
References

Scope and purpose

Sampling is a means of selecting a subset of units from a target population for the purpose of collecting information. This information is used to draw inferences about the population as a whole. The subset of units that are selected is called a sample. The sample design encompasses all aspects of how to group units on the frame, determine the sample size, allocate the sample to the various classifications of frame units, and finally, select the sample. Choices in sample design are influenced by many factors, including the desired level of precision and detail of the information to be produced, the availability of appropriate sampling frames, the availability of suitable auxiliary variables for stratification and sample selection, the estimation methods that will be used and the available budget in terms of time and resources.

Principles

There are two types of sampling: non-probability and probability sampling. Non-probability sampling uses a subjective method of selecting units from a population, and is generally fast, easy and inexpensive. Therefore, it's sometimes useful to perform things like preliminary studies, focus groups or follow-up studies. However, in order to make inferences about the population, one must make the often false assumption that the sample is representative. Probability sampling is based on three basic principles that make up the statistical framework. First, it is based on randomization, i.e. the units in the sample are selected at random. Second, all survey population units have a known positive probability of being selected in the sample, and third, we can calculate those probabilities, which are then used to calculate estimates along with estimates of the sampling error. The ability to make reliable inferences about the entire population and to quantify the error in the estimates makes probability sampling the best choice for most statistical programs.

The sample design should be as simple as possible. The aim is to produce estimates that are both precise and accurate enough to meet survey requirements. Precision is measured by the variance of an estimator. Lack of accuracy manifests itself through biases which are often introduced via non-sampling factors such as inaccurate reporting, processing, and measurement, as well as errors from nonresponse and incomplete reporting.

Guidelines

Design

• When determining sample size, take into account the required levels of precision needed for the survey estimates, the type of design (e.g., clustering, stratification) and estimator to be used, the availability of auxiliary and contact information, budgetary constraints, as well as other factors, such as nonresponse, presence of out-of-scope units, attrition in longitudinal surveys, etc. For periodic surveys, take into account expected births and deaths of units within the changing survey population. It's worth noting that the precision of survey estimates is usually influenced more by the total sample size than by the sampling fraction (ratio of the sample size to the population size).

• It is important to remember that most surveys produce estimates for many different variables, and optimizing the sample for one particular variable may have detrimental effects on other important variables. Handle this problem by first identifying the most important variables and then using this subset of variables to determine the sampling strategy to be adopted, which often requires a compromise between optimal strategies for the variables in the subset. See Bethel (1989).

• Stratification consists of dividing the population into subsets (called strata). Within each stratum, an independent sample is selected. The choice of strata is determined by the objectives of the survey, the availability of variables on the frame, the distribution of the variable of interest, and the desired precision of the estimates. Most surveys produce estimates for various domains of interest (e.g., provinces). If feasible, take this into account in the design by stratifying appropriately (e.g., by province). Otherwise, it will be necessary to consider special methods at the estimation stage to produce estimates for these domains (see Imputation). To achieve statistical efficiency, create strata in such a way that each stratum contains units that are as homogeneous as possible with respect to the information collected in the survey. For longitudinal surveys, choose stratification variables that correspond to characteristics that are stable over time.

• Conduct studies to evaluate alternative sampling methods, stratification options and allocation schemes. The usefulness of these studies depends on the availability and vintage of data used to conduct the studies, whether from previous censuses, surveys or administrative data and their relation to the variables of importance to the survey. See Kish (1988).

• Establish an expected response rate using a pre-test or data from previous occasions of the same or similar surveys. This rate can in turn be used in sample size determination. A sample can be divided into waves and additional waves of sample can be released as needed based on the achieved sample by stratum. For longitudinal surveys, expected cumulated attrition for the given number of cycles must be used.

Methods

• For highly skewed populations, create a stratum of large units to be included in the survey with certainty (the so-called take-all stratum). These large units would normally account for a significant portion of the population totals. In order to reduce respondent burden, the creation of a stratum of very small units to be excluded from the surveyed population is sometimes appropriate. See Baillargeon et al. (2007). It is important to distinguish between the non-surveyed portion of the survey population (the take-none stratum) which is part of the survey population but not sampled and the out-of-scope units which are not part of the survey population. The contribution for the take-none stratum may be estimated using models. 

• Sometimes the information needed to stratify the population is not available on the frame. In such cases, a two-phase (or double) sampling scheme may be used, whereby a large sample is selected in the first phase to obtain the required stratification information. This first sample is then stratified and in the second phase, a subsample is selected from each stratum within the first-phase sample. Consider the cost of sampling at each phase, the availability of the information required at each phase, and the gain in precision obtained by stratifying the first-phase sample.

• In practice, it is sometimes not feasible to directly select and contact the units that will report the requested information, due to either cost or lack of information. In such cases, a two-stage sampling scheme may be used by first selecting clusters (called primary sampling units)of reporting units, and then further subsampling within each of the selected primary sampling units to obtain a sample of reporting units. Budgetary or other constraints may necessitate more than two stages (a multi-stage design). Determine how many stages of sampling are needed and which sampling units are appropriate at each stage. For each possible type of units, consider the availability of a suitable frame of such units at each stage or the possibility of creating such a frame for the survey, ease of contact and of data collection/measurement, the quality of the data provided by the units, and the cost of collection. Multi-stage designs are by definition clustered sample designs. Clustering reduces data collection cost but may result in increased variances due to intra-cluster correlation. 

• If samples are to be selected from two or more frames, care must be taken in dealing with units that belong to more than one frame. It is important that the frame membership of each unit can be determined. The principle that the sample design should be simple is especially important when multiple frames are used. Choose sample designs that will simplify estimation procedures.

• In determining sample allocation and size for stratified samples, account for expected rates of misclassification of units and other deficiencies on the frame. If not properly considered at the sampling stage, survey estimates will not be as precise as planned. Address this problem at the estimation stage (see Section 2.10).

• In some complex sample designs, one must establish a design effect (DEFF) in order to determine the sample size. Results from previous or similar surveys should be used in calculating DEFF. See Gambino (2001), Kish (1965) and Gabler et al. (2006).

• For more complex situations, for example, when surveying rare or mobile populations, or when the sampling frame is a list of units that are linked, but don't correspond directly, to the units of the target population, special designs may be necessary. Such techniques as indirect sampling, network sampling or adaptive cluster sampling, to name a few, may need to be considered. See Lavallée (2007) and Thompson and Seber (1996).

• Random Digit Dialling (RDD) is a technique that has been popular for some types of household survey. RDD schemes have the potential for bias because there are households that do not have a landline telephone. With the increasing prevalence of cell-phone-only households, this problem will become more acute unless RDD starts being used for cell phone numbers as well. The potential biases need to be studied carefully before choosing the RDD option for a survey.

Periodic surveys

• For periodic surveys that use designs in which the sample size grows as the population increases, it is often appropriate to develop a method to keep the sample size and therefore collection costs, stable. One approach is to use a random drop method to maintain a stable sample size over time.

• For periodic surveys, make the design as flexible as possible to deal with future changes, such as increases or decreases in sample size, restratification, resampling and updating of selection probabilities. If estimates are required for specified domains of interest (e.g., subprovincial estimates), form the strata by combining small stable units related to the identified domains (e.g., small geographical areas), if possible. Future changes in definitions of the strata will then be easier to accommodate.

• For periodic surveys, if efficient estimates of change are required or if response burden is a concern, use a rotation sampling scheme that replaces part of the sample in each period. The choice of the rotation rate will be a compromise between the precision required for the estimates of change and the response burden on the reporting units. Lowering the rotation rate will increase the precision of the estimates of change but may lower the response rate over time because of increased response burden. A low rotation rate has the additional benefit of reducing costs if the first contact is substantially more expensive than subsequent contacts.

• For periodic surveys, develop procedures to monitor the quality of the sample design over time. Set up an update strategy for the selective redesign of strata that have suffered serious deterioration because of uneven growth.

Longitudinal surveys

• For longitudinal panel surveys, determine the length of the panel (its duration of time in the sample) by balancing the need for duration data with sample attrition and conditioning effects. Use a design with overlapping panels (i.e.with overlapping time span) when there is a need to produce cross-sectional estimates along with the longitudinal ones.

• It is particularly important to choose simple design features (i.e., one single frame) with as few stages or phases as possible as the estimation procedures becomes extremely complex with the number of waves.

• It is recommended that longitudinal surveys should be designed primarily to produce longitudinal estimates. Trying to satisfy both cross-sectional and longitudinal requirements can result in very complex design and estimation procedures. If production of cross-sectional estimates is required, the use of a "top-up" sample to cover births and new immigrants is recommended.

Implementation

• At the implementation stage, compare the size and characteristics of the actual sample to what was expected. Compare the precision of the estimates to the planned objectives. Reassess the assumptions used at the design stage. For example, evaluate nonresponse (non-contacts, refusals, etc.) and calculate design effects.

• Where possible, use generalized sample selection software instead of tailor-made systems. One such system is the Generalized Sampling System (GSAM) developed by Statistics Canada. GSAM is especially useful for managing sample selection and rotation for periodic surveys. By using generalized systems, one can expect fewer programming errors, as well as some reduction in development costs and time.

Documentation

• Write detailed and thorough documentation of all aspects of the sample design: which frames were chosen and why, how units were formed, how they were stratified, how sample size was determined and how sample was allocated, choice of stages and/or phases, which sampling schemes were used and why, and so on.

Quality indicators

Main quality element: accuracy

In addition to the points below, the reader should consult Statistics Canada Policy on informing users of data quality and methodology, which includes information that is pertinent here, particularly section E.1, subsection 2.3.

• Provide measures of the representativity of the sample: overcoverage and undercoverage, exclusions, comparisons to external sources (e.g. compare external demographic totals to those obtained from the survey).

• Compare the sample size observed to the expected sample size. This is especially important for multistage surveys where the sample size below the first stage may be hard to predict accurately.

• Compare response, attrition and out-of-scope rates to those that were assumed at the planning stage.

• Provide measures of sampling error: produce variances and/or CVs and compare them to the values expected at the planning stage; if design effects were used at the planning stage, compare them to realized design effects.

• For variables that were used to stratify the frame and/or allocate the sample, compare their observed CVs to their target values from the design stage.

• If possible, compare the observed homogeneity of strata to their homogeneity when they were formed; for a repeated survey, track the deterioration of the strata over time. Measure the frequency of stratum jumpers and of classification errors.

References

Bethel, J. 1989. "Sample allocation in multivariate surveys." Survey Methodology. Vol. 15, no. 1. p. 47-57.

Cochran, W.G. 1977. Sampling Techniques. New York. Wiley. 428 p.

Gambino, J. 2001. Design Effect Caveats. Internal document. Statistics Canada.

Gabler, S., S. Hader and P. Lynn. 2006. "Design Effects for Multiple Design Samples." Survey Methodology. Vol. 2, no. 1. p. 115-120.

Hidiroglou, M.A., 1994. "Sampling and estimation for establishment surveys: stumbling blocks and progress." Proceedings of the Section on Survey Research Methods. American Statistical Association. p. 153- 162.

Hidiroglou, M.A. and K.P. Srinath. 1993. "Problems associated with designing sub-annual business surveys." Journal of Business and Economic Statistics. Vol. 11. p. 397-405.

Kalton, G. and C.F. Citro. 1993. "Panel surveys: adding the fourth dimension." Survey Methodology. Vol. 19, no. 2. p. 205-215.

Kish, L. 1965. Survey Sampling. New York. Wiley-Interscience. 664 p.

Kish, L. 1988. "Multi-purpose Sample Designs." Survey Methodology. Vol. 14, no. 1. p. 19-32.

Lavallée, P. 2007. Indirect Sampling. New York. Springer. 256 p.

Lohr, S. 1999. Sampling: Design and Analysis. California. Duxbury Press, 512 p.

Särndal, C.-E., B. Swensson and J. Wretman. 1992. Model Assisted Survey Sampling. New York. Springer-Verlag. 694 p.

Statistics Canada. 2008. Methodology of the Canadian Labour Force Survey. Statistics Canada Catalogue no. 71-526-X. Ottawa, Ontario. 116 p.

Statistics Canada. 2003. Survey Methods and Practices. Statistics Canada, Catalogue no. 12-587-XPE. Ottawa, Ontario. 396 p.

Thompson, S.K. and G.A. Seber. 1996. Adaptive Sampling. New York. John Wiley and Sons. 288 p.

Tillé, Y. 2001. Théorie des sondages – Échantillonnage et estimation en populations finies. Paris, Dunod.