4. Simulations

Yan Lu

Previous | Next

In this section, a small simulation has been conducted to study the proposed chi-squared tests under a simple hypothesis $H_0\text{: }{q}_{ia}=p_{ia}{N}_a/N_A=q_{ia0}^A\mathrm{,}$ $q_{iab}^A=p_{iab}{N}_{ab}/N_A=q_{iab0}^A\mathrm{,}$ $q_{iab}^B=p_{iab}{N}_{ab}/N_B=q_{iab0}^B\mathrm{,}$ $q_{ib}=p_{ib}{N}_b/N_B=p_{ib0}$ to investigate the performance of chi-squared tests proposed in Section 3. We compare the percentages of samples for which the test statistics exceed the critical value to the nominal level $(\alpha =0.05).$ R (www.r-project.org) is used to perform simulation study and other analysis.

We generated the data following Skinner and Rao (1996), with ${\gamma }_a=N_a/N$ and ${\gamma }_b=N_b/N.$ A cluster sample from frame $A$ was generated with $n_p$ psus and $m$ observations in each psu, and a simple random sample of $n_B$ observations was generated for frame $B.$ We generated the clustered binary responses for the sample from frame $A$ by generating correlated multivariate normal random vectors and then using the probit function to convert the continuous responses to binary responses. After the sample was generated, we calculated the PML estimators of $p_{id}{N}_d/N_A$ and $p_{id}{N}_d/N_B$ (see Section 2.2). These estimated proportions were used to compute the chi-squared test statistics. We then compared the percentages of samples for which the test statistics exceed the critical value to the nominal level under different settings.

The simulation study was performed with factors: (1) ${\gamma }_a:0.4,$ (2) ${\gamma }_b:0.2,$ (3) clustering parameter $\rho :0.3,$ (4) sample sizes: $n_p:$ 10, 30 or 50; $m:$ 3, 5, or 10, $n_B:$ 100, 300 or 500. (5) Simulation runs: 1,000 times for each setting and 100 times when estimating the variance covariance matrix $V$ using bootstrapping. All runs used probability parameters $p_a\text{: }\left(\mathrm{.3,.1,.2,.4}\right),$ $p_{ab}\text{: }\left(\mathrm{.3,.1,.1,.5}\right),$ and $p_b\text{: }\left(\mathrm{.4,.1,.1,.4}\right).$ Table 4.1 reported the percentages of samples for which the test statistics exceed the critical value.

Table 4.1
Comparison of the actual significance levels (%) among different tests. $X^2$ is the uncorrected test; $X_{FC}^2$ is the first order corrected $X^2$ and $X_{SC}^2$ is the second order corrected $X^2$
Table summary
This table displays the results of Comparison of the actual significance levels (%) among different tests. $X^2$ is the uncorrected test; $X_{FC}^2$ is the first order corrected $X^2$ and $X_{SC}^2$ is the second order corrected $X^2$ . The information is grouped by ${\tilde{n}}_p$ (appearing as row headers), $n_B$ and Wald (appearing as column headers).

${\tilde{n}}_p$	$m$	$n_B$	$X^2$	Wald	$X_{FC}^2$	$X_{SC}^2$
10	3	100	12.1	17.3	5.6	4.9
30	3	300	13.6	8.4	4.8	4.8
50	3	500	15.5	10.0	6.4	3.6
10	5	100	25.7	13.5	7.5	4.9
30	5	300	29.2	9.3	7.9	5.3
50	5	500	31.5	8.5	8.1	4.9
10	10	100	46.1	21.2	6.6	5.4
30	10	300	50.2	11.5	7.5	5.6
50	10	500	58.7	8.0	9.6	5.1

Table 4.1 indicates that naively using uncorrected $X^2$ test for complex survey data is dangerous. With increased psu size and number of psu’s, the actual significance level even reaches 62.2%. Extended Wald test doesn’t perform well since the estimate of the variance may be unstable. Extended first order corrected test is acceptable with actual significance level around 7%. Extended second order corrected tests almost reach the nominal level 5%, for which is the one we recommend to use in a dual frame survey categorical data analysis.

Previous | Next