Primary mental health care visits in self-reported data versus provincial administrative records

Warning View the most recent version.

Archived Content

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please "contact us" to request a format other than those available.








by JoAnne L. Palin, Elliot M. Goldner, Mieke Koehoorn and Clyde Hertzman

General practitioners (GPs) are the main source of mental health care for most Canadians.  Information about the number of times individuals talk with GPs about mental health concerns is used for a variety of purposes,1,2 such as assessing adherence to treatment follow-up guidelines.  The major sources of population-level data on primary mental health care visits are surveys, particularly the 2002 Canadian Community Health Survey:  Mental Health and Well-being (CCHS 1.2),3,4 and provincial administrative health records, which the Public Health Agency of Canada now uses for national surveillance of "treated" mental health issues. Yet evidence from two previous studies5,6 indicates that survey data provide higher within-individual estimates of the number of mental health visits than do provincial administrative data.

One study5 found that the mean difference in the number of mental health visits to "health professionals" in 1994/1995 National Population Health Survey data was 28% higher than the number of mental health visits to "physicians" in Ontario Health Insurance Program data.  However, that study did not directly compare the number of mental health visits to GPs in each data source.  In addition, it focused on visits involving mental heath "services" (such as psychotherapy) in the administrative data and did not count general office visits for mental health reasons.

Another method for identifying mental health visits in administrative data is based on the diagnostic codes recorded as the reasons for the visits.7,8 Using the diagnostic code method (which is also used for national surveillance), Palin et al. 6 compared mental health visits in administrative data from the British Columbia Medical Services Plan with estimates from the 2002 CCHS in a linked sample. Because the CCHS asked about visits to GPs and to other types of health professionals separately, a direct comparison of GP visits in each data source was possible.  Among the respondents who had a mental health visit with a GP according to both data sources, the mean and median numbers of visits were higher in the self-reported data.  The study focused on the "main GP" seen for mental health issues in the past 12 months because the CCHS question referred to the GP with whom the respondent talked  "the most often."

It is possible that the diagnostic code method did not capture every visit during which the patient's mental health was discussed.  In British Columbia, GPs submit a single diagnostic code with each billing claim. However, primary care patients can present with a myriad of physical, emotional, social and personal issues,9 and physical symptoms may arise from mental health issues or vice versa.  If a GP recorded a diagnostic code for a physical condition or for "general symptoms" as the reason for the visit in a billing record, that visit would not be counted as a mental health visit in the administrative data, but could be counted in the self-reported survey data if mental health issues had been discussed.  In such cases, it may seem as if a survey respondent "over-reported" the number of mental health visits when compared with administrative data. 

Thus, the present study used a two-stage approach to count mental health visits in administrative data.  In the first stage, the number of mental health GP visits that CCHS respondents reported having had with the GP seen most often in the previous 12 months was compared with the number of visits to the main GP that were accompanied by a mental diagnostic code in the British Columbia Medical Services Plan data. 

In the second stage, the self-reported CCHS data were compared with the total number of visits to the main GP, regardless of diagnosis, in the administrative data.  If the number of self-reported mentalhealth visits to the main GP exceeded the total number of visits to the main GP in the administrative data, the difference between the data sources was not solely attributable to diagnostic coding.  These comparisons were made for the sample overall and by mental disorder. 

Data and methods

Data sources

The self-reported data are from a subsample of the cross-sectional 2002 Canadian Community Health Survey (CCHS) (cycle 1.2).  The survey covered people aged 15 or older living in private dwellings in the 10 provinces; it excluded residents of the territories, institutions, Indian reserves and certain remote areas, as well as members of the regular Armed Forces and civilian residents of military bases.  To obtain adequate sample sizes for young people and seniors, people aged 15 to 24 and 65 or older were over-sampled.  

The response rate to the CCHS in British Columbia was 77.7%.  For the respondents in the linked study sample, 86.2% of interviews were face-to-face; the others were conducted by telephone.  Detailed information about the survey design, methodology and questionnaire has been published elsewhere.3,4 Proxy interviews were not conducted. 

Administrative data are from the British Columbia Medical Services Plan database, which contains records of payments to physicians for medical services provided under fee-for-service arrangements.10

Data linkage and sample size

The 2002 CCHS sample for British Columbia totalled 3,902 respondents.  The British Columbia Ministry of Health Services linked the CCHS data to the Medical Services Plan data for those CCHS respondents who had given permission to do so, and who had provided their Personal Health Number (PHN).  The PHN is a unique identifier for individuals eligible to receive provincial health care services.  The Centre for Health Services and Policy Research (now Population Data BC), which maintains the data for research purposes, provided the administrative data extracts.  To ensure confidentiality, PHNs and physician billing numbers were replaced with anonymous study identifiers, and other potentially identifying information was removed.  The linkage was verified by comparing sex and month and year of birth in both data sources; 2,660 individuals met all the criteria.

The study reference period for each respondent was the 12 months before his/her survey interview date, so the data sources were compared for identical 12-month periods for each respondent individually.   For example, the study reference period might be March 3, 2001 to March 2, 2002 for one respondent, and March 15, 2001 to March 14, 2002 for another. 

The linked study sample consisted of 2,378 individuals who were continuously registered in the Medical Services Plan throughout the fiscal years pertaining to the study reference period.  The sample was reduced by 12 CCHS respondents who did not report the number of visits or who had in-patient care.  This yielded 2,366 individuals, representing 60.6% of the original CCHS sample for British Columbia.  The analytic subsample consisted of the 145 respondents who had primary mental health care according to both data sources (Table 1).

Table 1 Percentage distribution of selected demographic and mental health characteristics in Canadian Community Health Survey (CCHS) Public Use Microdata File, linked study sample and analytic subsample, household population aged 15 or older, British Columbia, 2002Table 1 Percentage distribution of selected demographic and mental health characteristics in Canadian Community Health Survey (CCHS) Public Use Microdata File, linked study sample and analytic subsample, household population aged 15 or older, British Columbia, 2002

Definition of GP visit

For this study, the number of self-reported mental health visits was obtained from the CCHS question:  "Think of the family doctor or the general practitioner you talked to the most often during the past 12 months.  How many times did you see, or talk on the telephone, to this family doctor or general practitioner (about your problems with your emotions, mental health or use of alcohol or drugs)?"  In this analysis, that doctor is considered the "main GP."

In the administrative data, a "visit" was defined as one or more Medical Services Plan claims for GP services for the same patient on the same date.  Diagnostic codes in the British Columbia Medical Services Plan data include ICD-9 codes, some ICD-9-CM (Clinical Modification) codes, and additional codes specific to the Medical Services Plan diagnostic coding system.7  The codes used to identify mental health visits were the Medical Services Plan diagnostic code 50B for "anxiety/depression," and ICD codes 290 to 314, and 316.  Anonymized physician identification numbers were used to determine which GP was seen most often by each respondent and to count the number of visits to that GP

Definition of mental disorder

Mental disorders (major depressive episode, manic episode, agoraphobia, panic disorder, social phobia) in the past 12 months were assessed in the 2002 CCHS with the World Mental Health version of the Composite International Diagnostic Interview that was adapted for the survey (CCHS 1.2/WMH-CIDI).4,11 Substance dependence (alcohol or illicit drugs) in the past 12 months was also assessed in the 2002 CCHS.4 Because of the small sample size, the three anxiety disorders were combined, and manic episodes were not included in the analysis.

Analytical techniques

The number of mental health visits to the main GP in the Medical Services Plan data was subtracted from the number of self-reported visits in the CCHS data for each CCHS respondent in the analytic subsample who had at least one visit according to both data sources.  These CCHS respondents were grouped into three categories:  High Reporters (self-reported mental health visits exceeded number in administrative data over same 12 months), Low Reporters (self-reported number was less than number in administrative data), and Exact Matches (self-reported visits equalled visits in administrative data).  Neither data source is considered to be the gold standard, so the labels of the three categories refer only to the direction of difference.

The chi-squared test of equal proportions was used to compare the percentages of High Reporters and Low Reporters with the percentage of Exact Matches.  Median and maximum differences between data sources were calculated to describe the magnitude of high and low reporting.

The percentages of High Reporters, Low Reporters and Exact Matches were also calculated for a second scenario:  the difference between the total number of visits to the main GP for any reason in the Medical Services Plan data and the number of self-reported mental health visits to the main GP in the CCHS data. 

Statistics Canada produces survey weights that correspond to the number of people in the population represented by a respondent.  For the vast majority of analyses, survey data should be weighted.  However, the aim of this study was to examine the actual level of case-by-case agreement between the two data sources before and after changing the definition of a primary mental health care visit in the administrative data. Therefore, population survey weights were not applied.12 Analyses were conducted using SPSS for Windows (Rel. 12.0.1. 2001. Chicago: SPSS Inc.).

Results

Scenario 1 (includes only mental health visits to main GP in administrative data)

Close to half (49%) of the survey respondents in the linked subsample were High Reporters; that is, in their CCHS interview, they reported more mental health visits to their main GP than were recorded in the Medical Services Plan data (Figure 1, Scenario 1). About a quarter (24%) were Low Reporters, and just over a quarter (27%) were Exact Matches. Most individuals in the subsample who had a mental disorder were High Reporters, with the percentage ranging up to 75%. By contrast, among the individuals in the subsample who did not meet the criteria for having had a mental disorder in the previous 12 months, the percentage of High Reporters (35%) did not exceed that of Exact Matches (38%), yet agreement remained low.

Figure 1 Percentage distribution of type of reporter of mental health visits to main General Practitioner in past 12 months, by definition of visit in administrative data and mental health status in past 12 months, household population aged 15 or older, British Columbia, 2002Figure 1 Percentage distribution of type of reporter of mental health visits to main General Practitioner in past 12 months, by definition of visit in administrative data and mental health status in past 12 months, household population aged 15 or older, British Columbia, 2002

For High Reporters, the median difference between the self-reported estimates and the administrative data was 3 visits, ranging from 2 visits for those with substance dependence to 4.5 visits for those with anxiety disorders (Table 2); the maximum difference was 42 visits. For Low Reporters, the median difference was 2 visits; the maximum difference, 36 visits.

Table 2 Difference in number of mental health visits to main General Practitioner (GP) in past 12 months in self-reported data versus administrative data, by mental health status in past 12 months and type of reporter, household population aged 15 or older, British Columbia, 2002Table 2 Difference in number of mental health visits to main General Practitioner (GP) in past 12 months in self-reported data versus administrative data, by mental health status in past 12 months and type of reporter, household population aged 15 or older, British Columbia, 2002

Scenario 2 (includes all visits to main GP in administrative data)

When the definition of a mental health visit in the administrative data was broadened to include all visits to the main GP, regardless of diagnostic codes, the percentage of Exact Matches was significantly higher for all mental health status categories (Figure 1, Scenario 2).  The overall percentage of High Reporters was halved from 49% to 25%, with a corresponding doubling of the percentage of Exact Matches from 27% to 51%.  Nonetheless, even under the broadened definition of a visit, only half the cases were Exact Matches.

Discussion

Survey respondents who had at least one mental health visit to a GP in the past 12 months according to both data sources, particularly those who had a measured mental disorder, tended to report more mental health visits per person than were recorded in administrative data.  This is consistent with previous research,5 although parallels are drawn cautiously because of differences in methodologies and data sources. 

Discrepancies between the two data sources were substantial.  For instance, CCHS respondents who met the criteria for depression reported as many as 19 more mental health visits to GPs than were found in their Medical Services Plan records, with a median difference of 4 visits.  Thus, in research in which the exact number of mental health visits to GPs is important, such as studies of adherence to treatment follow-up guidelines, the two data sources could potentially yield different results. 

Agreement between the two sources improved when the definition of a mental health visit in the administrative records was broadened to include all visits to the main GP.  Even so, one-quarter of individuals in the linked subsample reported more mental health visits than the total number of visits to the main GP for any reason in their administrative records.  It is possible that these respondents did not distinguish between GPs; in fact, according to administrative records, more than two-thirds of the 145 individuals in the sample saw more than one GP during the 12-month reference period (data not shown).  For 28% of the High Reporters, the number of self-reported mental health visits to the main GP was higher than the total number of visits to all GPs for any diagnosis in the administrative data, which could suggest respondent over-reporting in some cases (data not shown). 

For a number of  reasons, some GP visits that CCHS respondents reported may not have appeared in the Medical Services Plan data.  It could be that the GPs were reimbursed under an Alternative Payment Plan arrangement.  As well, reflecting the broad nature of the CCHS questions, respondents might have counted mental health contacts not reimbursed by the Medical Services Plan, such as telephone calls, which would not have been captured in the administrative data.  The extent to which these factors influenced the observed prevalence of High Reporting is not known, but evidence suggests that it would have been low.  According to the Canadian Institute for Health Information, more than 80% of physician payments in British Columbia were made on a fee-for-service basis (which appear in the Medical Services Plan database) during the fiscal years covering the study period.13 Moreover, when CCHS respondents were asked to select from a list of possible settings where their GP contacts took place, only one High Reporter (1.4%) selected "telephone consultation only," and no respondents in the analytic sample selected a location outside a health care setting (such as home, school, work, church or "other").

Even if improvements to either or both data sources increased the comparability of estimates, it may not be realistic to expect perfect agreement between data sources designed for different purposes and offering different perspectives—that of consumers and that of providers.  Analyses using both data sources, linked or in parallel, would enable data users to capitalize on the strengths and perspectives offered by each, and to evaluate the sensitivity of the results to the data source used.

Qualitative research would be useful to obtain information about diagnostic coding of mental health issues by GPs and by medical office staff, in order to understand if and why over- or under-representation of mental health visits may occur in administrative data. More complex algorithms using additional sources of administrative information about each individual, such as medical prescription data, could provide additional insight into the nature of medical visits.

Qualitative research would also be helpful in assessing potential causes of misreporting, such as the methods used by respondents to estimate the number of visits; difficulty recalling which visits occurred within the study reference period and which ones pertained to mental health; classifying other health professionals as a GP or family doctor; or misunderstanding the question. The possibility that respondents' mental health status affected their ability to accurately recall and report their visits should also be considered.

The findings of this analysis do not necessarily apply to other types of health professionals.  For example, in British Columbia, some psychiatrist visits may not appear in the Medical Services Plan data because psychiatrists are reimbursed on a fee-for-service basis to a lesser extent than are GPs.

Although the characteristics of the individuals in the linked study sample were similar to the characteristics of individuals in the complete 2002 CCHS Public Use Microdata File (PUMF) for British Columbia (Table 1), unmeasured psychological characteristics of non-linkers may have affected their recall and reporting of mental health visits. Compared with the full PUMF or full linked samples, a higher percentage of respondents in the analytic subsample (at least one mental health visit in both data sources) had a mental disorder or were in the low-income category, and fewer were younger than 25 or older than 65. 

This study focuses on methodological issues surrounding estimates of the number of mental health visits per person.  Consequently, the comparisons were made for people who had at least one primary mental health visit in both data sources. These findings cannot be generalized beyond their intended scope; other research on this sample has examined differences among people who had mental health care according to one data source but not the other.6

The CCHS asked about mental health visits to the mainGP.  Self-reported data were not available about the total number of visits to allGPs per person, for mental health issues or for other health issues.

Conclusion

Data linkage studies such as this provide opportunities to examine individual-level agreement between self-reported data and provincial administrative health records.  This is the first known study to compare the number of mental health visits reported to the Canadian Community Health Survey: Mental Health and Well-being with provincial administrative records, using diagnostic codes to identify visits in the administrative data, a method that was recently adopted by the Public Health Agency of Canada for national surveillance.  The study also involves a second stage of analysis using a broader definition encompassing "possibly" mental health-related visits (any diagnosis) and includes analyses of individuals who had anxiety disorders and/or substance dependence, in addition to those who had depression.

Although the median number of mental health visits to GPs in the self-reported and administrative data did not vary greatly for the total analytic sample (3 visits versus 2 visits), substantial per-person variations emerged between the data sources.  High Reporting was common, particularly among individuals with mental disorders, and the size of the difference was not trivial.  Even when all GP visits in the administrative data were counted, only half of the cases had exact agreement.  As well, one-quarter of the individuals in the sample were Low Reporters. 

Data users may find it beneficial to conduct parallel analyses with each data source, or to use linked data, in order to get a more complete picture of possible mental health care use.

Acknowledgements

This research was supported by a grant from the Canadian Institutes of Health Research (CIHR), and by training fellowships for JoAnne Palin from the CIHR; the Western Region Training Centre for Health Services Research (funded by the Canadian Health Services Research Foundation, the Alberta Heritage Foundation for Medical Research and the CIHR); the CIHR Strategic Training Program in Research in Addictions and Mental Health Policy and Services; and the British Columbia Health Research Foundation.  The authors gratefully acknowledge the statistical advice provided by Dr. Bruno Zumbo at the University of British Columbia.