2. Bagging survey estimators

Jianqiang C. Wang, Jean D. Opsomer and Haonan Wang

Previous | Next

2.1 General approach

In this section, we discuss the implementation of bagging in the context of survey estimation. We first introduce necessary notation. Let $U$  represent a finite population of size $N,$  in which each element $i\in U$  is associated with a vector of measurements, $y_i$ , in the $q$  -dimensional Euclidean space ${ℝ}^q.$  The sampling design $p\left(\right)$  is used to draw a random sample $A\subseteq U$  of sample size $n.$  We denote by $Y=\left\{y_i|i\in A\right\}$  the collection of sample observations. Here, the sampling design could be simple random sampling without replacement (SRSWOR), Poisson sampling or a complex design with stratification and/or multiple stages. Under each design, the probability of an element $i$  being included in the sample is denoted by ${\pi }_i.$

The population mean of the measurement vector $y$  is denoted by $\mu \text{.}$  It can be estimated by the Horvitz-Thompson (HT) estimator defined as

$$\widehat{\mu }=\frac{1}{N}{\displaystyle \sum _{i\in A}}\frac{y_i}{{\pi }_i}=\frac{1}{N}{\displaystyle \sum _{i=1}^N}\frac{y_i}{{\pi }_i}{I}_i\mathrm{,}(2.1)$$

where $I_i$  is the sample membership indicator for the $i\text{-th}$  element. More generally, let $\theta$  denote a population quantity of interest, and $\widehat{\theta }\left(Y\right)$  is the estimator of $\theta$  based on the sample observations $Y$ . The estimator $\widehat{\theta }\left(Y\right)$  will be abbreviated as $\widehat{\theta }$  when there is no confusion. As noted in the previous section, we assume that $\widehat{\theta }$  can be written as a function of simpler estimators of the form (2.1).

In its most general form, the bagging algorithm for survey estimation is as follows:

1. For $b=1,2,\dots \mathrm{,}B:$
   1. Draw resample $A_b$  from the random sample $A,$  and denote the observations in the resample as $Y_b^{*}=\left\{y_i|i\in A_b\right\}.$
   2. Calculate the parameter estimate based on the resample $A_b,$  denoted by $\widehat{\theta }\left(Y_b^{*}\right).$
2. Average over the replicated estimates $\widehat{\theta }\left(Y_1^{*}\right)\mathrm{,}\widehat{\theta }\left(Y_2^{*}\right)\mathrm{,}\cdots \mathrm{,}\widehat{\theta }\left(Y_B^{*}\right)$  to obtain the bagged survey estimator,

   $${\widehat{\theta }}_{bag}=\frac{1}{B}{\displaystyle \sum _{b=1}^B}\widehat{\theta }\left(Y_b^{*}\right)\mathrm{.}(2.2)$$

In the bagging literature, the resamples $A_b$  are often referred to as bootstrap samples (Breiman 1996), and we will do the same here despite the fact that we will not use them for variance estimation.

In the algorithm, the bootstrap samples could be drawn according to the sampling design rather than the empirical distribution of the sample observations, which is more commonly used in the ordinary bagging literature (Breiman 1996) and equivalent to simple random sampling (with or without replacement). For example, if the sample $A$  is drawn using stratified or cluster sampling, such design scheme could be taken into consideration when selecting the resamples. More generally in the survey context, step 1 of the proposed bagging algorithm can be treated in the framework of two-phase sampling: the first phase corresponds to the original sample $A$  and the second phase to the resample $A_b.$  Thus the classical expansion estimator for two-phase designs Särndal, Swensson and Wretman (1997) is implemented in calculating the replicated estimator $\widehat{\theta }\left(Y_b^{*}\right).$  In the resample $A_b,$  the pseudo inclusion probability for the $i\text{-th}$  element is ${\pi }_i^{*}={\pi }_i{\pi }_{i|A}$  where ${\pi }_{i|A}=\mathrm{Pr}\left(i\in A_b|i\in A\right)$  is the inclusion probability of the $i\text{-th}$  element in resample $A_b$  given that it is included in sample $A.$  Hence, the bagged estimator is an approximation to the expectation of the two-phase estimator with respect to the second sampling phase, which is also referred to as bootstrap expectation in ordinary bagging methods (Bühlmann and Yu 2002). Although a general design for the bootstrap samples is possible, in the theoretical portions of this article we will restrict ourselves to SRSWOR. To broaden the scope of our discussion, in the variance estimation and numerical section, we introduce the case in which the bootstrap samples are drawn by stratified SRSWOR with the same strata as the original sample $A,$  which is a useful and realistic extension.

As an example, we consider the HT estimator as defined in (2.1). The bootstrap resampling from the realized sample $A$  is drawn under SRSWOR of size $k.$  Under this resampling scheme, the replicated sample estimator is defined as

$$\widehat{\mu }\left(Y_b^{*}\right)=\frac{1}{N}{\displaystyle \sum _{i\in A_b}}\frac{y_i}{{\pi }_i^{\ast }}\mathrm{,}(2.3)$$

where the pseudo inclusion probability ${\pi }_i^{\ast }={\pi }_i{\pi }_{i|A}=k{\pi }_i/n.$  Then the bagged version of the classical ${\pi }^{\ast }$ -estimator can be calculated using (2.2). Straightforward calculation shows that the bagged estimator is identical to the original HT estimator if all SRSWOR samples of size $k$  are enumerated in calculating (2.2). The same result holds for any other linear survey estimator. In general, the calculation of the bagged estimator ${\widehat{\theta }}_{bag}$  is not as easy. In the rest of this section, we will focus on such calculations for the three types of nonlinear survey estimators discussed in Section 1.

2.2  Bagging differentiable survey estimators

For the survey estimators that are differentiable functions of HT estimators, the population quantity of interest can also be written as a differentiable function of population means; that is, ${\theta }_d=m\left(\mu \right),$  where $m(\cdot )$  is a known differentiable function. The subscript “$d$” stands for differentiable in contrast to non-differentiable $\left({\theta }_{nd}\right)$  and estimating equation $\left({\theta }_{ee}\right)$  coming later. A direct plug-in estimator of ${\theta }_d,$  based on sample observations $Y$ , can be written as

$${\widehat{\theta }}_d=m\left(\widehat{\mu }\right)\mathrm{,}(2.4)$$

where $\widehat{\mu }$  is defined in (2.1). Thus, the replicated sample version of ${\widehat{\theta }}_d$  can be expressed as

$${\widehat{\theta }}_d\left(Y_b^{*}\right)=m\left(\widehat{\mu }\left(Y_b^{*}\right)\right)\mathrm{,}$$

where $\widehat{\mu }\left(Y_b^{*}\right)$  is defined by (2.3). Then the bagged estimator of ${\theta }_d,$  denoted by ${\widehat{\theta }}_{d\mathrm{,}bag},$  is defined using (2.2).

2.3  Bagging explicitly defined non-differentiable estimators

As an example of this type of estimators, consider the estimation of the proportion of households with income below the poverty line for a population. Such quantity can be written as $\left(1/N\right){\displaystyle {\sum }_{i=1}^{N}I\left(y_i\le {\lambda }_N\right)},$  where $y_i$  is the income value for the $i\text{-th}$  household in the population, and ${\lambda }_N$  is the population poverty line. It can be seen that this quantity of interest is the mean of indicator kernel functions, and the kernel function is non-differentiable with respect to ${\lambda }_N.$  Here, we consider a more general class in which the kernel is an arbitrary non-differentiable but bounded function. This type of population quantity can be expressed as

$${\theta }_{nd}=\frac{1}{N}{\displaystyle \sum _{i=1}^N}h\left(y_i-{\lambda }_N\right)\mathrm{,}$$

where ${\lambda }_N$  is an unknown population parameter, for example, the mean, a quantile or other population quantity, and $h\left(y-\lambda \right)\text{:}{ℝ}^p\to ℝ$  is a non-differentiable function of $\lambda .$  The population quantity ${\theta }_{nd}$  generalizes the notion of the proportion below an estimated level and resembles the general form of a U-statistic.

Wang and Opsomer (2011) studied a class of U-statistics-like estimators, namely, non-differentiable survey estimators,

$${\widehat{\theta }}_{nd}=\frac{1}{N}{\displaystyle \sum _{i\in A}}\frac{1}{{\pi }_i}h\left(y_i-\widehat{\lambda }\right)\mathrm{,}(2.5)$$

where $\widehat{\lambda }$  is a design-based estimator of ${\lambda }_N.$  In the non-survey context, estimators of this type are regarded as “non-differentiable functions of the empirical distribution” (Bickel, Götze and van Zwet 1997). The study of appropriate bootstrap procedures for such estimators was carried out by Beran and Srivastava (1985) and Dümbgen (1993), among others. We define the replicated version of ${\widehat{\theta }}_{nd}$  based on resample $A_b$  as

$${\widehat{\theta }}_{nd}\left(Y_b^{*}\right)=\frac{1}{N}{\displaystyle \sum _{i\in A_b}}\frac{1}{{\pi }_i^{\ast }}h\left(y_i-\widehat{\lambda }\left(Y_b^{*}\right)\right)\mathrm{,}$$

where $\widehat{\lambda }\left(Y_b^{*}\right)$  solely depends on the bootstrap resample $A_b,$  and the bagged estimator is then defined by averaging replicated estimators. Suppose that the resampling process is SRSWOR of size $k,$  and every subsample is selected in calculating the bagging estimator, then the bagging estimator takes the following form after manipulation,

$${\widehat{\theta }}_{nd\mathrm{,}bag}=\frac{1}{N}{\displaystyle \sum _{i\in A}}\frac{1}{{\pi }_i\left(\begin{array}{c}n-1\\ k-1\end{array}\right)}{\displaystyle \sum _{A_b∍i}}h\left(y_i-\widehat{\lambda }\left(Y_b^{*}\right)\right)\mathrm{,}(2.6)$$

which replaces $h\left(y_i-\widehat{\lambda }\right)$  in (2.5) by a “smoothed” quantity ${\displaystyle \sum _{A_b∍i}}h\left(y_i-\widehat{\lambda }\left(Y_b^{*}\right)\right)/\left(\begin{array}{c}n-1\\ k-1\end{array}\right),$  by averaging the “jumps” in the estimator. Very often, variance reduction can be achieved by this replacement. The summand ${\displaystyle \sum _{A_b∍i}}h\left(y_i-\widehat{\lambda }\left(Y_b^{*}\right)\right)/\left(\begin{array}{c}n-1\\ k-1\end{array}\right)$  is the bootstrap expectation of $h\left(y_i-\cdot \right)$  and can be approximated using the convolution of $h\left(y_i-\cdot \right)$  with the sampling distribution of $\widehat{\lambda }\left(Y_b^{*}\right).$  Study of the theoretical aspects of ${\widehat{\theta }}_{nd\mathrm{,}bag}$  is deferred until Section 3.

2.4  Bagging estimators defined by non-differentiable estimating equations

Finally, we explain how to bag estimators defined by non-differentiable estimating equations. For ease of presentation, we consider a one-dimensional parameter of interest. The population parameter ${\theta }_{ee}$  of interest is defined as

$${\theta }_{ee}=\inf \left\{\gamma :S\left(\gamma \right)\ge 0\right\}\mathrm{,}$$

where

$$S\left(\gamma \right)=\frac{1}{N}{\displaystyle \sum _{i=1}^N}\psi \left(y_i-\gamma \right)\mathrm{,}$$

and $\psi (\cdot )$  is a non-differentiable real function. We can estimate the population parameter ${\theta }_{ee}$  by ${\widehat{\theta }}_{ee},$  where

$${\widehat{\theta }}_{ee}=\inf \left\{\gamma :\widehat{S}\left(\gamma \right)\ge 0\right\}$$

with

$$\widehat{S}\left(\gamma \right)=\frac{1}{N}{\displaystyle \sum _{i\in A}}\frac{1}{{\pi }_i}\psi \left(y_i-\gamma \right)\mathrm{.}$$

A frequently encountered estimator of this type is the sample quantile defined by inverting the sample cumulative distribution function (Francisco and Fuller 1991), where $\psi \left(y_i-\gamma \right)=I_{\left(y_i\le \gamma \right)}-\alpha$  for the $\alpha$-quantile.

Conceptually, there are two versions of bagging ${\widehat{\theta }}_{ee},$  one is to solve the “bagged estimating equation” defined by bagging the score function, and another is to average over resampled estimates of ${\widehat{\theta }}_{ee}.$  Similarly to the discussion in Section 2.1, the first version results in an estimator equivalent to the original estimator, because the bootstrap expectation of bootstrap samples of $\widehat{S}\left(\gamma \right)$  is equal to $\widehat{S}\left(\gamma \right)$  for fixed $\gamma .$  We therefore only consider the latter version. To define the bagged estimating equation estimator, we first define the replicated score function ${\widehat{S}}_b\left(\gamma \right)$  based on resample $A_b$  as

$${\widehat{S}}_b\left(\gamma \right)=\frac{1}{N}{\displaystyle \sum _{i\in A_b}}\frac{1}{{\pi }_i^{\ast }}\psi \left(y_i-\gamma \right)\mathrm{.}$$

Then the replicated estimator based on $A_b$  is defined as ${\widehat{\theta }}_{ee}\left(Y_b^{*}\right)=\inf \left\{\gamma :{\widehat{S}}_b\left(\gamma \right)\ge 0\right\}.$  Thus the overall bagging estimator is defined as

$${\widehat{\theta }}_{ee\mathrm{,}bag}=\frac{1}{\left(\begin{array}{c}n\\ k\end{array}\right)}{\displaystyle \sum }{\widehat{\theta }}_{ee}\left(Y_b^{*}\right)\mathrm{,}(2.7)$$

where the average is over all possible without-replacement samples of size $k$  selected from $A.$  Chen and Hall (2003) discussed bagging estimators defined by nonlinear estimating equations under the $iid$  setup, and they stated that bagging does not always improve the precision of estimators under study.

Previous | Next