5. Some simulations

Phillip S. Kott and Dan Liao

Previous | Next

Paralleling Kott and Liao (2012), we generated a synthetic population, $U,$  of hospitals from the 2008 DAWN public-use file. After creating $U,$  we independently drew 3,600 stratified simple random samples of size 400 from $U$  using the strata definitions on the public-use file. These definitions incorporate information on location and hospital ownership (public or private) not directly provided on the file.

We set the stratum sample sizes roughly proportional to a size measure $q_k,$  but never less than four. For $q_k$  we used annual drug-related emergency-room visits, which was always positive. The DAWN actually has a size variable attached to every hospital in the frame: total emergency-room visits in a previous year according to the American Hospital Association. Unfortunately, it was not included on the public-use file. Design weights in our simulations varied between 4.375 and 48, which allowed us to treat the finite population correction factors as ignorable in variance estimation.

As in our original paper, we generated a respondent sample $R$  for each simulated sample based on Bernoulli draw from the logistic function:

$$p_k={\left(1+\exp \left(\text{3}\text{.735}-\text{0}\text{.4}\log \left(q_k\right)\right)\right)}^{-1},(5.1)$$

We also created alternative respondent samples using

$$p_k={\left(1+\exp \left(\text{0}\text{.597}-\text{0}\text{.005}{q}_k^{1/2}\right)\right)}^{-1}.(5.2)$$

Both response models produce unweighted overall response rates of around 54%, which is similar to actual DAWN experience, where response is also a mildly increasing function of the size variable. Notice that ${\alpha }_k=1/p_k$  is bounded even if neither probability can be expressed by equation (2.4) with a finite $u.$

As in the previous study, we focused on estimating population totals for three survey variables. Annual drug-related emergency-room visits with adverse pharmaceutical reaction and those resulting in deaths came from the public-use file. Since both these variables were roughly linear in our size measure, the third “survey” variable was artificially constructed. It was the size measure (annual drug-related emergency-room visits) raised to the 1.3 power.

We investigated eight estimators and estimates of their variance. These are summarized in Table 5.1. The first two featured calibration to the original sample only (equation (2.5) with $\theta =1),$  with response assumed to be logistic in the log of the size measure. That is to say, equation (2.3) was employed with $x_k={\left(\text{1 }\log \left(q_k\right)\right)}^T.$  The first estimator used $z_k={\left(\text{1 }\log \left(q_k\right)\right)}^T$  as the calibration vector while the second used $z_k={\left(\text{1 }{q}_k\right)}^T,$  which was more consistent with a reasonable prediction model, at least for adverse reactions and deaths.

Our third and fourth estimator featured calibration to the sample and population in a single step (equation (2.5) with $\theta =1$  and then $\theta =0)$  using $x_k=z_k={\left(\text{1 }\text{l}\text{o}\text{g}\left(q_k\right)q_k\right)}^T.$  They were designed to be nearly unbiased if either the logistic response model in ${\left(\text{1 }\log \left(q_k\right)\right)}^T$  or the linear prediction model in ${\left(\text{1 }{q}_k\right)}^T$  held.

Not surprisingly, the (empirical) relative mean squared error of the fourth estimator is always lower than the third. The reason is fairly obvious looking at equation (3.1) and considering the consequence of $\theta$  being 0 (calibration to the population) rather than 1 (calibration to the sample).

The fifth through eighth estimators were calibrated in two steps. The fifth and seventh estimators employed the calibration weighting from the first estimator in its first step, while the sixth and eighth employed the calibration weighting from the second estimator. The fifth and sixth used $z_{2k}=x_{2k}={\left(\text{1 }\log \left(q_k\right)q_k\right)}^T$  in their second step, while the seventh and eighth were nearly pseudo-optimal (Kott 2011) using $z_{2k}={\left(\text{1 }\log \left(q_k\right)q_k\right)}^T$  and $x_{2k}=\left(d_k{\alpha }_k-1\right)z_{2k}$  in their second step. All four employed the individual weight-adjustment functions:

$$h_k\left(g_2^T{x}_{2k}\right)=\frac{1}{d_k{\alpha }_k}+\left(1-\frac{1}{d_k{\alpha }_k}\right)\exp \left[\frac{g_2^T{x}_{2k}}{1-\frac{1}{d_k{\alpha }_k}}\right].$$

As Kott (2011) showed these $h_k\left(g_2^T{x}_{2k}\right)$  are asymptotically identical to the weight-adjustment function, $1+g_2^T{x}_{2k},$  when $g_2^T{x}_{2k}={\text{O}}_P\left(1/\sqrt{n}\right)$  but prevent any $w_k$  from falling below unity. Each is a version of equation (4.1) with ${\ell }_k=1/\left(d_k{\alpha }_k\right),c=1,$  and $u=\infty .$

Because the nonresponse rate was so large, we did not encounter a problem computing the third and fourth estimator using any of the simulated respondent samples. The relative mean squared error of the fourth estimator was always slightly higher than that of the seventh and eighth estimators, which incorporated nearly pseudo-optimal calibration in their second step. Interestingly, this was not the case when comparing the fourth estimator to the fifth and sixth estimators which, although employing two steps, did not incorporate nearly pseudo-optimal calibration.

Observe that although the second estimator always had a smaller relative mean squared error than the first, being more consistent with a reasonable prediction model (even for $q_k^{\text{1}\text{.3}},$  the survey variable appeared closer to being linear in $q_k$  than in $\log \left(q_k\right)),$  the other analogous pairs (fifth vs sixth and seventh vs eighth) exhibited no clear pattern of superiority. This is because it is the second-step residuals that are effectively modeled in equation (4.4) not the $y-$ values.

Generating the nonresponse with equation (5.2) than (5.1) did not seem to have much of an impact on the results except for the relative biases of the first estimator. For both adverse reactions and ${\left(\text{size}\right)}^{1.3},$  the relative bias of this estimator is over 40% of the relative mean squared error. That is likely because both models that could be used to justify this estimator (response is logistic in the log of the size measure and the survey variable is linear in the log of the size measure) fail. Not surprisingly, since the relative bias is such a large part of the relative mean squared error in these two situations, $v\left(t_k\right)$  underestimates mean squared error badly. Nowhere else is the relative bias of $v\left(t_k\right)$  greater than 15%.

It seems that even our artificial variable, ${\left(\text{size}\right)}^{1.3},$  was close enough to being linear in the size measure that bias was never an issue for any estimator other than the first. The first estimator itself had a negligible relative bias when response was a logistic model of the log of the size measure, as assumed.

Previous | Next