Weighting and estimation
A typical survey objective is to estimate descriptive population parameters, as well as analytical parameters, on the basis of a sample selected from a population of interest. Examples of parameters include simple descriptive statistics such as totals, means, ratios and percentiles. Examples of analytical parameters include regression coefficients, correlation coefficients and measures of income inequality.
In a probability-based survey, a design weight is associated with each sampled unit. The design weight can be interpreted as the number of typical units in the survey population that each sampled units represents. Estimates can be calculated using the design weights or estimation weights obtained by adjusting the design weights. Common adjustments include those that account for nonresponse and that incorporate auxiliary information. See Statistics Canada (2003).
The precision of an estimate is an important aspect of quality. This aspect is measured using the estimated standard error (square root of the estimated variance). Improvements to this precision can be obtained by incorporating auxiliary data into the estimation process.
In a probability-based survey, all elements of the population have a known probability of being selected into the sample. These inclusion probabilities take into account aspects of the sample design such as stratification, clustering and multi-stage or multi-phase selection. The design weight is equal to the inverse of the inclusion probability in single-phase (one stage) sampling. It is the product of the inverse selection probabilities at each phase (stage) in a multi-phase (multi-stage) design.
If there is unit nonresponse, the observed sample is smaller in size than the original sample selected. To compensate for unit nonresponse reweighting can be performed by adjusting the design weights. These adjustment factors should be based on each unit's probability of response, which can be estimated using models.
If auxiliary data are available, some improvement in the precision of estimates may be achieved. Incorporation of auxiliary data in the estimation process is known as calibration. Calibration consists of adjusting the weights such that estimates of the auxiliary variable(s) satisfy known totals (also referred to as control totals). Calibration includes well-known estimators such as the regression, the ratio and the raking-ratio estimators (Deville and Särndal, 1992). Desirable properties of calibration include:
Coherent estimates between different sources;
Potential improvements to the precision of estimates;
Potential reduction of unit nonresponse error and coverage error.
Estimates are produced by summing the data multiplied by either the design or estimation weights. There are two types of errors associated with these estimates: sampling and non-sampling errors. The sampling error is the error caused by observing a sample instead of the whole population (Särndal et al., 1992). It is measured by the sampling variance, which depends on the sample design, and any auxiliary data that are used in the estimation procedure. Non-sampling errors include coverage (imperfect frame), measurement, processing and nonresponse errors.
An estimate of the sampling variance can be calculated using methods such as Taylor linearization or resampling methods such as the jackknife and the bootstrap. Regardless of which method is used, it has to incorporate sample design properties such as stratification, clustering and multi-stage or multi-phase selection, if applicable.
It is more difficult to measure nonsampling errors. They may require additional data that are typically not available. Examples include repeated measures to evaluate measurement errors and recontact of nonrespondents to evaluate nonresponse bias.
A weight needs to be associated with each sampled unit. This weight can be the design weight or the estimation weight (e.g., calibration weight). If only the design weight is used, the resulting estimator is called the Horvitz-Thompson estimator. If auxiliary information is used by calibration, the resulting estimator is called the calibration estimator. The weight associated with this estimator is known as the estimation or calibration weight. An estimation weight should be used whenever the design weight has been adjusted for nonresponse or auxiliary data.
As full response is unlikely, nonresponse adjustments should be used to minimize the nonresponse bias. Applying these adjustments within subsets of the population can minimize the nonresponse bias. It is assumed that the non-respondents behave similarly to the respondents within these subsets. These subsets are delineated using auxiliary information (Lundström and Särndal, 2005) or propensity models (Eltinge and Yansaneh, 1997).
If there are auxiliary data that are correlated with the variable(s) of interest, then calibration must be considered. These auxiliary data must be at least available for the sampled units, and the corresponding population totals must be known. The resulting calibration estimator will usually have a smaller variance than the Horvitz-Thompson estimator. In addition, the weighted auxiliary data will add up to the population totals.
Calibration weights can be very large, or even negative. If this occurs, methods exist to control the range of the weights. See Huang and Fuller (1978) or Deville and Särndal (1992).
Composite estimation should be considered for periodic surveys with a large sample overlap between occasions. It is a calibration method that treats the data from previous occasions as auxiliary variables. For more details, see Gambino, Kennedy and Singh (2001).
Two sets of weights can be associated with longitudinal surveys: longitudinal weights and cross-sectional weights. Longitudinal weights refer to the population at the initial selection of the longitudinal sample. For longitudinal analysis, these weights should be adjusted to account for sample attrition. Cross-sectional weights reflect the population at a point in time. These weights can be used to produce point estimates or differences between time periods.
If double (two-phase) sampling has occurred, weighting needs to reflect the design and any auxiliary data available for the population or the first phase sample.
The estimation process must use the estimation weights to compute descriptive and analytical statistics for domains of interest. The estimation weights are equivalent to the design weights if no adjustments have been made. Corresponding variance estimators must reflect the sample design, any adjustments to the design weights, imputation and the estimation method. Variances can be estimated using linearization methods or resampling methods (jackknife, balanced repeated replication, and bootstrap). For more details, see Wolter (2007).
Small domains refer to subpopulations where there is not enough sample (or even no sample at all) to produce reliable estimates. It is therefore reasonable to incorporate the requirements of these domains at the sample design stage (Singh, Gambino and Mantel, 1994). If this is not possible at the design stage, or if the domains are only specified at a later stage, consider special estimation methods (small area estimators) at the estimation stage. These methods "borrow strength" from related areas (or domains) to minimize the mean squared error of the resulting estimator (Rao, 2003).
Generalized estimation software should be used when appropriate (Estevao et al., 1995).
Main quality element: accuracy
The quality of a point estimate is usually described in terms of accuracy and precision. Accuracy represents how closely a measured value agrees, on average, with the true value. The accuracy of an estimator is gauged in terms of how close the average of its realized values is to the parameter of interest. This is measured by comparing its design expectation to the parameter, and the difference is called the bias. Precision, on the other hand, refers to how closely individual measurements agree with each other. Precision is usually measured by the sampling error, the error that results from observing a sample instead of the whole population. If an estimator is unbiased, its mean squared error is equal to its sampling variance.
Unbiased estimators should be used if they exist and are efficient in terms of variance. Slightly biased estimators can be used if their efficiency, measured in terms of mean squared error, is smaller than the variance of corresponding unbiased estimators.
The coefficient of variation is normally used to describe the precision of an estimate. It is defined as the standard error of the estimate divided by the true value of the parameter. An estimate with a given coefficient of variation is less precise than one with a smaller coefficient of variation. Because of potential division by zero, as well as interpretation problems, the use of coefficients of variation should be restricted to positive variables of interest. Otherwise, standard errors should be used.
Estimators that incorporate auxiliary data assume that the models between the target variables and auxiliary data hold for all units in the population. In practice, however, it is difficult to determine whether model assumptions are valid. Estimates that use auxiliary data should be accompanied by a description of the model assumptions made and an assessment of the likely effect of making these assumptions on the quality of estimates.
Deville, J.-C. and C.E. Särndal. 1992. "Calibration estimators in survey sampling." Journal of the American Statistical Association. Vol. 87. p. 376-382.
Eltinge, J.L. and I.S. Yansaneh. 1997. "Diagnostics for formation of nonresponse adjustment cells, with an application to income nonresponse in the U.S. Consumer Expenditure Survey." Survey Methodology. Vol. 23. p. 33-40.
Estevao, V., M.A. Hidiroglou and C.E. Särndal. 1995. "Methodological principles for a generalized estimation system at Statistics Canada." Journal of Official Statistics. Vol. 11, p. 181-204.
Gambino, J., B. Kennedy and M.P. Singh. 2001. "Regression composite estimation for the Canadian Labour Force Survey: Evaluation and implementation." Survey Methodology. Vol. 27, no. 1. p. 65-74.
Huang, E. T. and W.A. Fuller. 1978. "Nonnegative regression estimation for sample survey data." Proceedings of the Social Statistics Section, American Statistical Association. p. 300-303.
Lundström, S. and C.-E. Särndal. 2005. Estimation in Surveys with Nonresponse. New York. John Wiley and Sons. 212 p.
Rao, J.N.K. 2003. Small Area Estimation. New York. John Wiley and Sons. 344 p.
Särndal, C.E., B. Swensson and J.H. Wretman. 1992. Model Assisted Survey Sampling. New York. Springer-Verlag. 694 p. Springer Series in Statistics.
Singh, M.P., J. Gambino. and H. Mantel 1994. "Issues and strategies for small area data." Survey Methodology. Vol. 20. p. 3-14.
Statistics Canada 2003. Survey methods and practices. Statistics Canada Catalogue no. 12-587-XPE, Ottawa, Ontario. 396 p.
Wolter, K. 2007. Introduction to Variance Estimation. 2nd ed. New York. Springer-Verlag. 449 p.
- Date modified: