3. Proposed method

Jae Kwang Kim and Shu Yang

Previous | Next

We first consider a particular fractional hot deck imputation method, called full fractional imputation, where the imputed values are taken from the set of respondents denoted as $A_R=\left\{i\in A;{\delta }_i=1\right\}$. That is, the $j\text{-th}$ imputed value of missing $y_i,$ denoted by $y_i^{*(j)},$ is equal to the $j\text{-th}$ value of $y$ among the set in $A_R.$ We propose a fractional hot deck imputation approach that makes use of the parametric model assumption $f\left(y|x;\theta \right)$. If all of the elements in $A_R$ are selected as the imputed values for missing $y_i,$ we can treat $\left\{y_j;j\in A_R\right\}$ as a realization from $f\left(y_j|{\delta }_j=1\right)$ and fractional weight assigned to donor $y_j$ for the missing item $y_i$ is, by choosing $h\left(y_j|x_i\right)=f\left(y_j|{\delta }_j=1\right)$ in (2.6),

$$\begin{array}{llllll}{w}_{ij}^{\mathrm{<mo>*</mo>}}\hfill & \propto \hfill & f\left(y_j\mathrm{<mo>|</mo>}{x}_i\mathrm{,}{\delta }_i\mathrm{<mo>=</mo>0<mo>;</mo>}\widehat{\theta }\right)/f\left(y_j\mathrm{<mo>|</mo>}{\delta }_j\mathrm{<mo>=</mo>1}\right)\hfill & \hfill & \hfill & (3.1)\hfill \\ \hfill & \propto \hfill & f\left(y_j\mathrm{<mo>|</mo>}{x}_{i\mathrm{<mo>;</mo>}}\widehat{\theta }\right)/f\left(y_j\mathrm{<mo>|</mo>}{\delta }_j\mathrm{<mo>=</mo>1}\right)\mathrm{,}\hfill & \hfill & \hfill & \hfill \end{array}$$

with ${\displaystyle {\sum }_{j;{\delta }_j=1}}{w}_{ij}^{*}=1$, and $\widehat{\theta }$ being the MLE obtained from (2.4). The second line follows from the MAR assumption. Furthermore, we can write

$$\begin{array}{llllll}f\left(y_j\mathrm{<mo>|</mo>}{\delta }_j\mathrm{<mo>=</mo>1}\right)\hfill & =\hfill & {\displaystyle \int }f\left(y_j|x\mathrm{,}{\delta }_j\mathrm{<mo>=</mo>1}\right)f\left(x|{\delta }_j\mathrm{<mo>=</mo>1}\right)dx\hfill & \hfill & \hfill & (3.2)\hfill \\ \hfill & =\hfill & {\displaystyle \int }f\left(y_j|x\right)f\left(x|{\delta }_j\mathrm{<mo>=</mo>1}\right)dx\hfill & \hfill & \hfill & \hfill \\ \hfill & \cong \hfill & \frac{1}{N_R}{\displaystyle \sum _{k\mathrm{<mo>=</mo>1}}^N}{\delta }_{k}f\left(y_j|x_k\right)\mathrm{,}\hfill & \hfill & \hfill & \hfill \end{array}$$

where the second equality follows from the MAR assumption, and the last (approximate) equality follows by approximating the integral by the population empirical distribution, and $N_R$ is the number of respondents in the population. Using the survey weights, we can approximate

$$f\left(y_j|{\delta }_j=1\right)\cong \frac{{\displaystyle {\sum }_{k\in A_R}{w}_{k}f\left(y_j|x_k\right)}}{{\displaystyle {\sum }_{k\in A_R}{w}_k}}$$

and the fractional weights in (3.1) are computed from

$$w_{ij}^{*}\propto \frac{f\left(y_j|x_i;\widehat{\theta }\right)}{{\displaystyle {\sum }_{k\in A_R}{w}_{k}f\left(y_j|x_k;\widehat{\theta }\right)}} (3.3)$$

with ${\displaystyle {\sum }_{j\in A_R}}{w}_{ij}^{*}=1$. In (3.3), the point mass $w_{ij}^{*}$ assigned to donor $y_j$ for missing unit $i$ is expressed by the ratio of the density $f\left(y|x\right).$ Thus, for each missing unit $i,n_R=\left|A_R\right|$ observations are used as donors for the hot deck imputation using $w_{ij}^{*}$ as the fractional weights. Such fractional imputation can be called full fractional imputation (FFI) because there is no randomness due to the imputation mechanism. The FFI estimator of $\eta ,$ defined by ${\displaystyle {\sum }_{i=1}^N}U\left(\eta ;x_i\mathrm{,}{y}_i\right)=0$, is then computed by solving

$${\displaystyle \sum _{i\in A}}{w}_i\left\{{\delta }_{i}U\left(\eta ;x_i\mathrm{,}{y}_i\right)+\left(1-{\delta }_i\right){\displaystyle \sum _{j\in A_R}}{w}_{ij}^{*}U\left(\eta ;x_i\mathrm{,}{y}_j\right)\right\}=\mathrm{0,} (3.4)$$

where $w_{ij}^{*}$ is defined in (3.3). Note that the imputed estimating equation (3.4) is a good approximation to the expected estimating equation in (2.2).

In survey sampling, an imputed data set with a large imputation size may not be desirable. Thus, instead of taking all the observations in $A_R$ as donors for each missing item, a subset of $A_R$ can be selected to reduce the size of the donor set of missing $y_i.$ Thus, the selection of the donors is viewed as a sampling problem and we use an efficient sampling design and weighting techniques to obtain efficient imputation estimators. For the donor selection mechanism, efficient sampling designs, such as a stratified sampling design or systematic Proportional-to-Size (PPS) sampling, can be used to select donors of size $m.$ A systematic PPS sampling for fractional hot deck imputation can be described as follows:

1. Within each $i$ with ${\delta }_i=0,$ sort the donors in the full respondent set $\left\{y_j;{\delta }_j=1\right\}$ in ascending order as $y_{(1)}\le \cdots \le y_{(r)}$ and use $w_{i(j)}^{*}$ to denote the fractional weight associated with $y_{(j)}.$ That is, $w_{i(j)}^{*}=w_{ik}^{*}$ for $y_{(j)}=y_k.$
2. Partition $\left[\mathrm{0,1}\right]$ by $\left\{I_j\equiv \left[{\displaystyle {\sum }_{k=0}^j}{w}_{i(j)}^{*}\mathrm{,}{\displaystyle {\sum }_{k=0}^{j+1}}{w}_{i(j)}^{*}\right)\mathrm{,}j=1,\dots \mathrm{,}r-1\right\},$ where $w_{i(0)}^{*}=0.$
3. Generate $u\sim \text{uniform}\left(\mathrm{0,}1/m\right)$ and let $u_k=u+k/m,$ $k=0,\dots \mathrm{,}m-1.$ For $k=0,\dots \mathrm{,}m-1,$ if $u_k\in I_j$ for some $0\le j\le r-1,$ include $j$ in the sample $D_i.$

After we select $D_i$ from the complete set of respondents, the selected donors in $D_i$ are assigned with the initial fractional weights $w_{ij0}^{*}=1/m$. The fractional weights are further adjusted to satisfy

$${\displaystyle \sum _{i\in A}}{w}_i\left\{\left(1-{\delta }_i\right){\displaystyle \sum _{j\in D_i}}{w}_{ij\mathrm{,}c}^{*}q\left(x_i\mathrm{,}{y}_j\right)\right\}={\displaystyle \sum _{i\in A}}{w}_i\left\{\left(1-{\delta }_i\right){\displaystyle \sum _{j\in A_R}}{w}_{ij}^{*}q\left(x_i\mathrm{,}{y}_j\right)\right\}\mathrm{,} (3.5)$$

for some $q\left(x_i\mathrm{,}{y}_j\right)$, and ${\displaystyle {\sum }_{j\in D_i}}{w}_{ij\mathrm{,}c}^{*}=1$ for all $i$ with ${\delta }_i=0,$ where $w_{ij}^{*}$ is the fractional weights for FFI method, as defined in (3.3). Regarding the choice of the control function $q\left(x\mathrm{,}y\right)$ in (3.5), we can use $q\left(x\mathrm{,}y\right)={\left(y\mathrm{,}{y}^2\right)}^{\prime }$, which keeps the empirical distributions of $y$ for $D_i$ and $A_R$ as close as possible in the sense that the first and second moment of $y$ are the same. Other choices can also be considered. See Fuller and Kim (2005).

The problem of adjusting the initial weights to satisfy certain constraints is often called calibration and the resulting fractional weights can be called calibrated fractional weights. Using the idea of regression weighting, the final calibration fractional weights that satisfy (3.5) and ${\displaystyle {\sum }_j}{w}_{ij\mathrm{,}c}^{*}=1$ can be computed by

$$w_{ij\mathrm{,}c}^{*}=w_{ij0}^{*}+w_{ij0}^{*}\Delta \left(q_{ij}^{*}-{\overline{q}}_{i\cdot }^{*}\right)\mathrm{,} (3.6)$$

where $q_{ij}^{*}=q\left(x_i\mathrm{,}{y}_j\right),{\overline{q}}_{i\cdot }^{*}={\displaystyle {\sum }_{j\in A_R}}{w}_{ij0}^{*}{q}_{ij}^{*},$

$\Delta ={\left\{C_q-{\displaystyle {\sum }_{i\in A}{w}_i\left(1-{\delta }_i\right)}{\displaystyle {\sum }_{j\in A_R}{w}_{ij0}^{*}{q}_{ij}^{*}}\right\}}^T{\left\{{\displaystyle {\sum }_{i\in A}{w}_i\left(1-{\delta }_i\right)}{\displaystyle {\sum }_{j\in A_R}{w}_{ij0}^{*}{\left(q_{ij}^{*}-{\overline{q}}_{i\cdot }^{*}\right)}^{\otimes 2}}\right\}}^{-1}$

and $C_q={\displaystyle {\sum }_{i\in A}}{w}_i\left\{\left(1-{\delta }_i\right){\displaystyle {\sum }_{j\in A_R}}{w}_{ij}^{*}q\left(x_i\mathrm{,}{y}_j\right)\right\}$. Here, $B^{\otimes 2}$ denotes $B{B}^T.$ Some of the fractional weights computed by (3.6) can take negative values. If that happens, algorithms alternative to regression weighting should be used. For example, consider entropy weighting, where the fractional weights of the form

$$w_{ij\mathrm{,}c}^{*}=\frac{w_{ij}^{*}\exp \left(\Delta q_{ij}^{*}\right)}{{\displaystyle {\sum }_{k\in A_R}{w}_{ik}^{*}\exp \left(\Delta q_{ik}^{*}\right)}} (3.7)$$

are approximately equal to the regression fractional weights in (3.6) and are always positive. Once the calibration fractional weights are obtained, the FHDI estimator of $\eta$ is then computed by solving

$${\displaystyle \sum _{i\in A}}{w}_i\left\{{\delta }_{i}U\left(\eta ;x_i\mathrm{,}{y}_i\right)+\left(1-{\delta }_i\right){\displaystyle \sum _{j\in D_i}}{w}_{ij\mathrm{,}c}^{*}U\left(\eta ;x_i\mathrm{,}{y}_j\right)\right\}=0. (3.8)$$

For variance estimation, a replication method can be used. See Appendix A.1 for a brief discussion of the replication variance estimator for the proposed method.

Furthermore, the proposed method can handle non-ignorable non-response under the correct specification of the response model. See Appendix A.3 for the extension to non-ignorable non-response case.

Previous | Next