7. Conclusions and discussion

Paul Knottnerus

Previous

This section summarizes a number of conclusions and issues for further research.

When totals of turnover are estimated from a panel in months $t$ and $t-12,$ two estimators ${\widehat{g}}_{STN}$ and ${\widehat{g}}_{OLP}$ for the growth rate between these months can be distinguished.

When using ${\widehat{g}}_{STN},$ one should be aware that in practice, $\mathrm{var}\left({\widehat{g}}_{OLP}\right)$ might be much smaller than $\mathrm{var}\left({\widehat{g}}_{STN}\right)$ especially when the turnover in month $t-12$ and the turnover in month $t$ are highly correlated and the overlap ratios $\lambda$ and $\mu$ are not too small.

The efficiency of ${\widehat{g}}_{STN}$ and ${\widehat{g}}_{OLP}$ can be improved by the composite estimator ${\widehat{g}}_{COM}$ described in Section 4.

Using least squares techniques, an aligned composite vector-estimator ${\left({\widehat{g}}_{AC},{\widehat{Y}}_{AC},{\widehat{X}}_{AC}\right)}^{\prime }$ can be derived that obeys the nonlinear restriction for totals and growth rates: ${\widehat{Y}}_{AC}=\left(1+{\widehat{g}}_{AC}\right){\widehat{X}}_{AC}.$

The AC estimates estimator subject to linear restrictions can be extended in several ways: (i) for nonlinear restrictions, (ii) for different data sets such as monthly, quarterly and yearly data, (iii) for births and deaths, (iv) for regression and ratio estimators, and (v) for additional auxiliary variables.

Similar to the regression estimator, the AC estimates estimator is asymptotically unbiased. This remark also applies to the covariance-matrix estimator $\left(I_k-\widehat{K}R\right){\widehat{V}}_0.$

There is not yet an unambiguous answer on the question of to what extent data from the past should be included in the vector estimate ${\widehat{\theta }}_0$ each month. The answer depends upon: (i) the NSI’s policy and rules with respect to revision of already published figures, (ii) the fact that from a theoretical viewpoint, the sequence of $T$ monthly SRS estimates ${\overline{y}}_1,{\overline{y}}_2,\mathrm{...},{\overline{y}}_T$ (included as component in ${\widehat{\theta }}_0)$ should be so long that the difference between the two AC estimates estimators of ${\overline{Y}}_1,$ say ${\widehat{\overline{Y}}}_{1AC}^T$ and ${\widehat{\overline{Y}}}_{1AC}^{T+1},$ is not substantial, and (iii) the size of the samples. That is, in analogy with the regression estimator or, equivalently, the calibration estimator, the sample sizes should be much larger than the number of (calibration) restrictions. For a simulation study on the variance of the regression estimator and the number of regressors, see Silva and Skinner (1997) and for a relationship between the regression estimator and the GR estimator, see Appendix A.3 and Knottnerus (2003).

In the specific case of estimating mutually aligned totals and changes, additional research is needed for finding: (i) the optimal and practical length for the monthly, quarterly, semesterly and yearly series of SRS estimates to be included in the initial vector ${\widehat{\theta }}_0$ and (ii) a rule of thumb with respect to the number of restrictions compared to the sample sizes in order to find an AC estimates estimator ${\widehat{\theta }}_{AC}$ with an improved efficiency.

Acknowledgements

The views expressed in this paper are those of the author and do not necessarily reflect the policy of Statistics Netherlands. The author would like to thank Harm Jan Boonstra, Arnout van Delden, Sander Scholtus, the Associate Editor and two anonymous referees for their helpful comments and corrections.

Appendix

A.1 Proofs of (3.4) and (4.5)

The proof of (3.4) is as follows. For $n_{12}=n_{23}=n,$ formula (2.2) can be rewritten as

$$\mathrm{var}\left({\widehat{g}}_{STN}\right)\approx \frac{1}{{\overline{X}}^2}\left\{\left(\frac{1}{n}-\frac{1}{N}\right)S_{y-Gx}^2+2\left(\frac{1}{n}-\frac{\lambda }{n}\right)G{S}_{xy}\right\}(\text{A}.1)$$

Dividing (2.4) by (A.1) yields

$$\begin{array}{lll}Q=\frac{\mathrm{var}\left({\widehat{g}}_{OLP}\right)}{\mathrm{var}\left({\widehat{g}}_{STN}\right)}\hfill & \approx \hfill & \frac{\left(\frac{1}{\lambda n}-\frac{1}{N}\right)S_{y-Gx}^2}{\left(\frac{1}{n}-\frac{1}{N}\right)S_{y-Gx}^2+2\left(\frac{1}{n}-\frac{\lambda }{n}\right)G{S}_{xy}}\hfill \\ \hfill & =\hfill & \frac{\left({\lambda }^{-1}-f\right)S_{y-Gx}^2}{\left(1-f\right)S_{y-Gx}^2+2\left(1-\lambda \right)G{S}_{xy}}\hfill \\ \hfill & =\hfill & \left({\lambda }^{-1}-f\right){\left(1-f+2\left(1-\lambda \right)\frac{G{S}_{xy}}{S_{y-Gx}^2}\right)}^{-1}\hfill \\ \hfill & \approx \hfill & \left({\lambda }^{-1}-f\right){\left(1-f+2\left(1-\lambda \right)\frac{{\rho }_{xy}^2}{1-{\rho }_{xy}^2}\right)}^{-1}.\text{(A}\text{.2)}\hfill \end{array}$$

In the last line we used that under the model assumptions mentioned in Section 3, $G{S}_{xy}\approx {\widehat{B}}^2{S}_x^2={\rho }_{xy}^2{S}_y^2$ and $S_{y-Gx}^2\approx \left(1-{\rho }_{xy}^2\right)S_y^2,$ provided that $N$ is sufficiently large; see also the derivation of (3.2).

Next, under the same assumptions, (4.5) can be derived as follows. Since $n_{12}=n_{23}=n,$ the covariance in (4.3) can be rewritten as

$$\mathrm{cov}\left({\widehat{g}}_{OLP},{\widehat{g}}_{STN}\right)\approx \frac{1}{{\overline{X}}^2}\left(\frac{1}{n}-\frac{1}{N}\right)S_{y-Gx}^2.(\text{A}.3)$$

Combining (2.4), (A.1) and (A.3), we can write $k$ in (4.2) as

$$\begin{array}{cc}k& \approx \frac{\left(\frac{1}{\lambda n}-\frac{1}{n}\right)S_{y-Gx}^2}{\left(\frac{1}{\lambda n}-\frac{1}{n}\right)S_{y-Gx}^2+2\left(\frac{1}{n}-\frac{\lambda }{n}\right)G{S}_{xy}}\hfill \\ & ={\left(1+\frac{2\lambda G{S}_{xy}}{S_{y-Gx}^2}\right)}^{-1}\approx {\left(1+\frac{2\lambda {\rho }_{xy}^2}{1-{\rho }_{xy}^2}\right)}^{-1}.\hfill \end{array}$$

Similar to deriving (A.2), we used in the last line $G{S}_{xy}/S_{y-Gx}^2\approx {\rho }_{xy}^2/\left(1-{\rho }_{xy}^2\right).$

A.2 Derivation of (5.5)

In case of $m$ linear restrictions $c-R\theta =0,$ matrix $K$ can be found by minimizing

$$\underset{K}{\min }E\left[{\left\{\theta -{\widehat{\theta }}_0-K\left(c-R{\widehat{\theta }}_0\right)\right\}}^{\prime }\left\{\theta -{\widehat{\theta }}_0-K\left(c-R{\widehat{\theta }}_0\right)\right\}\right];$$

see Knottnerus (2003, page 330). The solution of this least squares problem is given by

$$\begin{array}{l}K=E\left\{\left(\theta -{\widehat{\theta }}_0\right){\left(c-R{\widehat{\theta }}_0\right)}^{\prime }\right\}{\left[\mathrm{cov}\left(c-R{\widehat{\theta }}_0\right)\right]}^{-1}(\text{A}.4)\\ =V_0{R}^{\prime }{\left(R{V}_0{R}^{\prime }\right)}^{-1}.\end{array}$$

In case of $m$ nonlinear restrictions, the new minimand is

$$E\left[{\left\{\theta -{\widehat{\theta }}_0-K\left[c-R\left({\widehat{\theta }}_0\right)\right]\right\}}^{\prime }\left\{\theta -{\widehat{\theta }}_0-K\left[c-R\left({\widehat{\theta }}_0\right)\right]\right\}\right].$$

Similarly to (A.4), it can be shown that this minimand attains its minimum for

$$K=E\left\{\left(\theta -{\widehat{\theta }}_0\right){\left[c-R\left({\widehat{\theta }}_0\right)\right]}^{\prime }\right\}{\left[\mathrm{cov}\left\{c-R\left({\widehat{\theta }}_0\right)\right\}\right]}^{-1}.(\text{A}.5)$$

Substituting Taylor’s linearization $R\left({\widehat{\theta }}_0\right)\approx R\left(\theta \right)+D_R\left(\theta \right)\left({\widehat{\theta }}_0-\theta \right)$ into (A.5), we get the following approximation, say $K_1,$ for $K$

$$\begin{array}{cc}{K}_1^{}& \approx V_0{D^{\prime }}_R\left(\theta \right){\left[D_R\left(\theta \right)V_0{D^{\prime }}_R\left(\theta \right)\right]}^{-1}\hfill \\ & \approx V_0{D^{\prime }}_R\left({\widehat{\theta }}_0\right){\left[D_R\left({\widehat{\theta }}_0\right)V_0{D^{\prime }}_R\left({\widehat{\theta }}_0\right)\right]}^{-1}.(\text{A}.6)\hfill \end{array}$$

Assuming that ${\widehat{\theta }}_0~N\left(\theta ,V_0\right),$ the first approximation for the constrained maximum likelihood (ML) solution, say ${\widehat{\theta }}_{ML}^{(1)},$ can be calculated in the standard manner by using the linearized restrictions

$${\widehat{\theta }}_{ML}^{(1)}={\widehat{\theta }}_0+K_1\left\{c\left({\widehat{\theta }}_0\right)-D_R\left({\widehat{\theta }}_0\right){\widehat{\theta }}_0\right\},(\text{A}.7)$$

where $c\left({\widehat{\theta }}_0\right)$ is defined by (5.4). If ${\widehat{\theta }}_{ML}^{(1)}$ does not satisfy the nonlinear restrictions $c-R\left(\theta \right)=0,$ a better approximation of $K$ might be obtained by replacing ${\widehat{\theta }}_0$ in (A.6) by update ${\widehat{\theta }}_{ML}^{(1)}$ resulting in a new matrix $K_2^{}.$ In turn, in analogy with (A.7) $K_2^{}$ leads to a better approximation or update of ${\widehat{\theta }}_0,$ say ${\widehat{\theta }}_{ML}^{(2)},$

$${\widehat{\theta }}_{ML}^{(2)}={\widehat{\theta }}_0+K_2\left\{c\left({\widehat{\theta }}_{ML}^{(1)}\right)-D_R\left({\widehat{\theta }}_{ML}^{(1)}\right){\widehat{\theta }}_0\right\},$$

where we used Taylor’s linearization of the nonlinear restrictions around $\theta ={\widehat{\theta }}_{ML}^{(1)}.$ Repeating this procedure, we get the following recursions for ${\widehat{\theta }}_{ML}^{(h)}$ or, for short, ${\widehat{\theta }}_h$

$$\begin{array}{l}{\widehat{\theta }}_h={\widehat{\theta }}_0+K_h\left\{c_h-D_h{\widehat{\theta }}_0\right\}\\ K_h=V_0{D^{\prime }}_h{\left[D_h{V}_0{D^{\prime }}_h\right]}^{-1}\left(h=1,2,\mathrm{...}\right)\text{.}\end{array}$$

For definitions of $c_h$ and $D_h,$ see Section 5; in practice, $V_0$ should be replaced by its estimate ${\widehat{V}}_0.$ By construction, for each $h$ we have

$$\begin{array}{cc}0& =c\left({\widehat{\theta }}_{ML}^{(h-1)}\right)-D_R\left({\widehat{\theta }}_{ML}^{(h-1)}\right){\widehat{\theta }}_{ML}^{(h)}\hfill \\ & =c-R\left({\widehat{\theta }}_{ML}^{(h-1)}\right)+D_R\left({\widehat{\theta }}_{ML}^{(h-1)}\right){\widehat{\theta }}_{ML}^{(h-1)}-D_R\left({\widehat{\theta }}_{ML}^{(h-1)}\right){\widehat{\theta }}_{ML}^{(h)};\hfill \end{array}$$

see (5.4). Hence, when ${\widehat{\theta }}_{ML}^{(h)}$ converges to the (constrained) maximum likelihood solution ${\widehat{\theta }}_{ML}^{},$ $c-R\left({\widehat{\theta }}_{ML}^{(h-1)}\right)$ converges to zero. Also, assuming $K_h^{}$ converges to say ${\widehat{K}}_{ML}^{},$ the corresponding covariance matrix of ${\widehat{\theta }}_{ML}^{},$ say $V_{ML},$ can be approximated by

$$V_{ML}^{}\approx \left\{I_k-K{D}_R\left(\theta \right)\right\}{V}_0,$$

which for sufficiently large $h$ can be estimated by ${\widehat{V}}_{ML}=\left(I_k-K_h{D}_h\right){\widehat{V}}_0;$ see also Cramer (1986, page 38).

A.3 Regression estimator as GR estimator

Suppose that $Y_i$ and the auxiliary variable $Z_i,$ with known population mean $\overline{Z},$ are observed in $s_2.$ In order to apply the GR estimator to this situation, define

$${\widehat{\theta }}_0=\left(\begin{array}{l}{\overline{y}}_2\\ {\overline{z}}_2\end{array}\right),V_0=\mathrm{cov}\left({\widehat{\theta }}_0\right)=\left(\frac{1}{n_2}-\frac{1}{N}\right)\left(\begin{array}{l}{S}_y^2{S}_{yz}\\ S_{yz}{S}_z^2\end{array}\right).$$

The prior restriction is

$$0=c-R\theta =\overline{Z}-(0,1)\left(\begin{array}{l}{\theta }_y\\ {\theta }_z\end{array}\right).$$

Applying (5.1) and (5.2) to this case yields the following GR estimator

$$\begin{array}{l}{\widehat{\theta }}_{GR}={\widehat{\theta }}_0+K\left(c-R{\widehat{\theta }}_0\right)=\left(\begin{array}{l}{\overline{y}}_2\\ {\overline{z}}_2\end{array}\right)+K\left(\overline{Z}-{\overline{z}}_2\right)\\ K=V_0{R}^{\prime }{\left(R{V}_0{R}^{\prime }\right)}^{-1}=\left(\begin{array}{l}{S}_{yz}\\ S_z^2\end{array}\right)\frac{1}{S_z^2}=\left(\begin{array}{l}{b}_{yz}\\ 1\end{array}\right)\left(b_{yz}=S_{yz}/S_z^2\right)\\ V_{GR}=\left(I_2-KR\right)V_0=\left(\begin{array}{l}1-b_{yz}\\ 0\text{ 0}\end{array}\right)V_0.\end{array}$$

Hence, replacing $b_{yz}$ by its estimate $b_{yz2}=s_{yz2}/s_{z2}^2,$ we can approximate the first element in ${\widehat{\theta }}_{GR}$ by ${\widehat{\theta }}_{GRy}\approx {\overline{y}}_2+b_{yz2}\left(\overline{Z}-{\overline{z}}_2\right)$ which corresponds to the familiar regression estimator, often denoted by ${\widehat{\overline{Y}}}_{REG}.$ For sufficiently large $n_2,$ the variance of ${\widehat{\overline{Y}}}_{REG}$ can be approximated by

$$\begin{array}{lll}\mathrm{var}\left({\widehat{\overline{Y}}}_{REG}\right)\hfill & \approx \hfill & \mathrm{var}\left({\widehat{\theta }}_{GRy}\right)={\left[V_{GR}\right]}_{11}=\left(\frac{1}{n_2}-\frac{1}{N}\right)\left(S_y^2-b_{yz}{S}_{yz}\right)\hfill \\ \hfill & =\hfill & \left(\frac{1}{n_2}-\frac{1}{N}\right)S_e^2;\text{(A}\text{.8) }\hfill \\ \hfill S_e^2& =\hfill & \frac{1}{N-1}{{\displaystyle \sum _{i\in U}\left\{Y_i-\overline{Y}-b_{yz}\left(Z_i-\overline{Z}\right)\right\}}}^2;\hfill \end{array}$$

recall from regression theory that $b_{yz}{S}_{yz}=b_{yz}^2{S}_z^2$ and $S_y^2=b_{yz}^2{S}_z^2+S_e^2.$ The variance in (A.8) can be estimated by the well-known variance estimator

$$\mathrm{v}\widehat{\mathrm{a}}\mathrm{r}\left({\widehat{\overline{Y}}}_{REG}\right)=\left(\frac{1}{n_2}-\frac{1}{N}\right)s_{\widehat{e}2}^2\text{, where }{s}_{\widehat{e}2}^2=\frac{1}{n_2-1}{\displaystyle \sum _{i\in s_2}{\left\{Y_i-{\overline{y}}_2-b_{yz2}\left(Z_i-{\overline{z}}_2\right)\right\}}^2}.$$

Similar results can be derived for more than one auxiliary variable. This illustrates once more that with respect to the bias and the variance approximation the AC estimates estimator strongly resembles the regression estimator or, equivalently, the calibration estimator.

References

Berger, Y.G. (2004). Variance estimation for measures of change in probability sampling. The Canadian Journal of Statistics, 32, 451-467.

Cochran, W.G. (1977). Sampling Techniques. New York: John Wiley and Sons, Inc.

Cramer, J.S. (1986). Econometric Applications of Maximum Likelihood Methods. Cambridge: Cambridge University Press.

Hidiroglou, M.A., Särndal, C.E. and Binder, D.A. (1995). Weighting and estimation in business surveys. In Business Survey Methods, (Eds., B.G. Cox et al.). New York: John Wiley and Sons, Inc.

Kalman, R.E. (1960). A new approach to linear filtering and prediction problems. Transactions ASME, Journal of Basic Engineering, 82, 35-45.

Kish, L. (1965). Survey sampling. New York: John Wiley and Sons, Inc.

Knottnerus, P. (2003). Sample Survey Theory: Some Pythagorean Perspectives. New York: Springer-Verlag.

Knottnerus, P. and Van Delden, A. (2012). On variances of changes estimated from rotating panels and dynamic strata. Survey Methodology, 38(1), 43-52.

Laniel, N. (1987). Variances for a rotating sample from a changing population. Proceedings of the Survey Research Methods Section, American Statistical Association, 496-500.

Magnus, J.R. and Neudecker, H. (1988). Matrix Differential Calculus with Applications in Statistics and Econometrics. New York: John Wiley and Sons, Inc.

Nordberg, L. (2000). On variance estimation for measures of change when samples are coordinated by the use of permanent random numbers. Journal of Official Statistics, 16, 363-378.

Qualité, L. and Tillé, Y. (2008). Variance estimation of changes in repeated surveys and its application to the Swiss survey of value added. Survey Methodology, 34(2), 173-181.

Särndal, C.E., Swensson, B. and Wretman, J. (1992). Model Assisted Survey Sampling. New York: Springer-Verlag.

Silva, P.L.D.N. and Skinner, C.J. (1997). Variable selection for regression estimation in finite populations. Survey Methodology, 23(1), 23-32.

Smith, P., Pont, M. and Jones, T. (2003). Developments in business survey methodology in the Office for National Statistics, 1994–2000. Journal of the Royal Statistical Society D, 52, 257–295.

Tam, S.M. (1984). On covariances from overlapping samples. The American Statistician, 38, 288–289.

Wood, J. (2008). On the covariance between related Horvitz-Thompson estimators. Journal of Official Statistics, 24, 53-78.

Previous