3. Markov models for contingency tables with nonresponse

Andrés Gutiérrez, Leonardo Trujillo and Pedro Luis do Nascimento Silva

Previous | Next

Consider the problem of estimating gross flows between two consecutive periods of time using categorical data obtained from a panel survey and under nonresponse. Also, suppose that the outcome of every interview is the classification of the respondent into any of $G$ possible pairwise disjoint categories, and the aim is to estimate the gross flows between these categories using the information from individuals who were interviewed at two consecutive periods of time. Individuals who either did not answer in one or two periods or were excluded or included for only one of the two periods shall not have a definite classification among the categories. Then, there is one group of individuals with classification between the two periods, a group of individuals who only have the information for one of the two periods and a group of individuals who did not respond in any of the two periods of the survey.

For those individuals responding on times $t-1$ and $t$, the classification data can be summarized in a matrix of dimension $G\times G$. The available information for those individuals not responding the survey at time $t-1$ but responding at time $t$ can be summarized in a column complement; the information for those individuals not responding at time $t$ but responding at time $t-1$ can be summarized in a row complement. Finally, individuals not responding at any of the two times are included in a single cell counting the number of individuals with missing data at both times.

The whole matrix is illustrated in Table 3.1, where $N_{ij}$ ( $i\mathrm{,}j=1,\dots \mathrm{,}G$ ) denotes the number of individuals in the population having classification $i$ at time $t-1$ and classification $j$ at time $t$, $R_i$ denotes the number of individuals not responding at time $t$ and having classification $i$ at time $t-1$, $C_j$ denotes the number of individuals not responding at time $t-1$ and had classification $j$ at time $t$, and $M$ denotes the number of individuals in the sample not responding in any of the two times. It is important to mention that this analysis does not take into account nonresponse due to the rotation in the survey; it only takes into account individuals belonging to the matched sample ignoring those individuals not responding because they were not selected in the sample.

Table 3.1
Gross flows at two consecutive periods of time.
Table summary
This table displays the results of Gross flows at two consecutive periods of time.. The information is grouped by Time $t-1$ (appearing as row headers), Time $t$ (appearing as column headers).

Time $t-1$	Time $t$
	1	2	$\cdots$	$G$	Row complement
1	$N_{11}$	$N_{12}$	$\cdots$	$N_{1G}$	$R_1$
2	$N_{21}$	$N_{22}$	$\cdots$	$N_{2G}$	$R_2$
$⋮$	$⋮$	$⋮$	$⋮$	$⋮$	$⋮$
$G$	$N_{G1}$	$N_{G2}$	$\cdots$	$N_{GG}$	$R_G$
Column complement	$C_1$	$C_2$	$\cdots$	$C_G$	$M$

This paper considers ideas from Stasny (1987) and Chen and Fienberg (1974) - in the sense of considering a maximum likelihood approach in contingency tables for partially classified data - and data resulting from a two-stage process as follows:

1. In the first stage (nonobservable), the individuals are located among the cells of a matrix $G\times G$ according to the probabilities of a Markov chain process. Let ${\eta }_i$ be the initial probability of an individual being at the category $i$ at the time $t-1$ with ${\displaystyle {\sum }_i}{\eta }_i=1,$ and $p_{ij}$ be the transition probability from the category $i$ to category $j$ , where ${\displaystyle {\sum }_j}{p}_{ij}=1$ for every $i$ .
2. In the second stage (observable) of the process, every individual in cell $ij$ can either be nonrespondent at time $t-1,$ losing the classification by row; nonrespondent at time $t,$ losing the classification by column; or nonrespondent at both times, losing both classifications.
   • Let $\psi$ be the initial probability of an individual in cell $ij$ responding at time $t-1.$
   • Let ${\rho }_{RR}$ be the transition probability of classification of the individual in cell $ij$ responding at time $t-1$ and responding at time $t.$
   • Let ${\rho }_{MM}$ be the transition probability of an individual in cell $ij$ being a nonrespondent at time $t-1$ and becoming a nonrespondent at time $t.$
   These probabilities do not depend on the classification stage of the individual.

Data is observed only after the second stage. The aim is to make inferences for the probabilities in the Markov chain process generating the data but also in the chain generating the nonresponse mechanism. In the context of this two-stage model, the corresponding probabilities are shown in Table 3.2.

Table 3.2
Gross flow probabilities at two consecutive times.
Table summary
This table displays the results of Gross flow probabilities at two consecutive times.. The information is grouped by Time $t-1$ (appearing as row headers), Time $t$ (appearing as column headers).

Time $t-1$	Time $t$
	1	2	$\cdots$	$j$	$\cdots$	$G$	Row complement
1
2
$⋮$
$i$	$\left\{{\eta }_i{p}_{ij}\psi {\rho }_{RR}\right\}$						$\left\{{\displaystyle {\sum }_j}{\eta }_i{p}_{ij}\psi \left(1-{\rho }_{RR}\right)\right\}$
$⋮$
$G$
Column complement	$\left\{{\displaystyle {\sum }_i}{\eta }_i{p}_{ij}\left(1-\psi \right)\left(1-{\rho }_{MM}\right)\right\}$						${\displaystyle {\sum }_i}{\eta }_i{p}_{ij}{\displaystyle {\sum }_j}\left(1-\psi \right){\rho }_{MM}$

In this way, the likelihood function for the observed data under this two-stage model is proportional to

$$\begin{array}{l}{\displaystyle \prod _i}{\displaystyle \prod _j}{\left[\psi {\rho }_{RR}{\eta }_i{p}_{ij}\right]}^{N_{ij}}\times {\displaystyle \prod _i}{\left[{\displaystyle \sum _j}\psi \left(1-{\rho }_{RR}\right){\eta }_i{p}_{ij}\right]}^{R_i}\\ \times {\displaystyle \prod _j}{\left[{\displaystyle \sum _i}\left(1-\psi \right)\left(1-{\rho }_{MM}\right){\eta }_i{p}_{ij}\right]}^{C_j}\times {\left[{\displaystyle \sum _i}{\displaystyle \sum _j}\left(1-\psi \right){\rho }_{MM}{\eta }_i{p}_{ij}\right]}^M.(3.1)\end{array}$$

3.1 Parameters of interest

Data are only observed after the second stage and the aim is to make inferences for both the probabilities at the Markov chain generating the data and the chain generating nonresponse. Under this two-stage model, the probabilities of the matrix of data are shown in Table 3.2 and they constitute some of the parameters of interest.

On the other hand, coming from the non-observable process, it is necessary to consider other parameters of interest as follows. Suppose a finite population $U$ exists, having a classification in two periods of time for all its individuals. This is a non-observable process as, even when census data is obtained, it would be not possible to have a complete classification since not all the individuals will be willing to respond. Considering this non-observable process and assuming that there are $G$ possible classifications at each time, the distribution of the gross flows at the population level are shown in Table 3.3.

$X_{ij}$ is the number of units at the finite population with classification $i$ at time $t-1$ and classification $j$ at time $t$ ( $i\mathrm{,}j=1,\dots \mathrm{,}G$ ). The population size, $N$, must satisfy the expression:

$$N={\displaystyle \sum _i}{\displaystyle \sum _j}{X}_{ij}.$$

Table 3.3
Population gross flows (non-observable process) at two consecutive periods of time.
Table summary
This table displays the results of Population gross flows (non-observable process) at two consecutive periods of time.. The information is grouped by Time $t-1$ (appearing as row headers), Time $t$ (appearing as column headers).

Time $t-1$	Time $t$
	1	2	$\cdots$	$j$	$\cdots$	$G$
1	$X_{11}$	$X_{12}$	$\cdots$	$X_{1j}$	$\cdots$	$X_{1G}$
2	$X_{21}$	$X_{22}$	$\cdots$	$X_{2j}$	$\cdots$	$X_{2G}$
$⋮$	$⋮$	$⋮$	$\ddots$	$⋮$	$\ddots$	$⋮$
$i$	$X_{i1}$	$X_{i2}$	$\cdots$	$X_{ij}$	$\cdots$	$X_{iG}$
$⋮$	$⋮$	$⋮$	$\ddots$	$⋮$	$\ddots$	$⋮$
$G$	$X_{G1}$	$X_{G2}$	$\cdots$	$X_{Gj}$	$\cdots$	$X_{GG}$

Following the non-observable process from the last section, it is supposed that the vector corresponding to the entries at the last contingency table follows a multinomial distribution with a probability vector containing the values ${\left\{{\eta }_i{p}_{ij}\right\}}_{i\mathrm{,}j=1,\dots \mathrm{,}G}$. This assumes a superpopulation model where the contingency table counts are considered random. In terms of notation, the probability measure considering these counts will be denoted with the subindex $\xi .$ Then, the probability of classification at cell $i\mathrm{,}j$ for the $k$ -th individual is

$$\begin{array}{l}{P}_{\xi }\left(k\text{ has got a classification }i\text{ at }t-1\text{ and classification }j\text{ at }t\right)\\ =P_{\xi }\left(k\text{ has got a classification }i\text{ at }t-1\right)\\ \times P_{\xi }\left(k\text{ has got a classification }j\text{ at }t|k\text{ has got a classification }i\text{ at }t-1\right)\\ ={\eta }_i{p}_{ij}.\end{array}$$

This treats $X_{ij}$ as a random variable and if the finite population has $N$ individuals, its expected value based on the model is given by

$$E_{\xi }\left(X_{ij}\right)=N{\eta }_i{p}_{ij}={\mu }_{ij}.(3.2)$$

Note that this expected value ${\mu }_{ij}$ is one of the most important parameters to be estimated on this paper as it corresponds to the expected value of the gross flows at the population of interest at the two consecutive periods. On the other hand, it is also important to understand that ${\mu }_{ij}$ is a parameter for the two-stage model. Also, the estimators for ${\eta }_i$ and $p_{ij}$ are interdependent and determined by the estimations of the defined parameters at the second stage. Let $\mathbf{\eta }$ be the vector containing the parameters ${\eta }_i$; and $p$ be the vector containing the parameters $p_{ij}$, for every $i\mathrm{,}j=1,\dots \mathrm{,}G$. The final parameters of interest are:

• the model parameters, determined by the vector

  $$\theta \mathrm{<mo>=</mo>}{\left({\psi }^{\prime }\mathrm{,}{{\rho }^{\prime }}_{RR}\mathrm{,}{{\rho }^{\prime }}_{MM}\mathrm{,}{\eta }^{\prime }\mathrm{,}{p}^{\prime }\right)}^{\prime };$$

• the expected value vector of the population counts defined as

  $$\mu \mathrm{<mo>=</mo>}{\left({\mu }_{11}\mathrm{,}\dots \mathrm{,}{\mu }_{ij}\mathrm{,}\dots \mathrm{,}{\mu }_{GG}\right)}^{\prime }.$$

Previous | Next