Comparability of self-reported medication use and pharmacy claims data

Warning View the most recent version.

Archived Content

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please "contact us" to request a format other than those available.

By Sara Allin, Ahmed M. Bayoumi, Michael R. Law and Audrey Laporte

Information on prescription drug use for population-level research is available from community surveys and from administrative data. While each data source offers advantages and disadvantages for the investigation of medication use, the two sources are rarely compared. The aim of this article is to examine the agreement between two sources of drug utilization data available for Ontario.

Numerous surveys have gathered information on medication use. The design and implementation of the surveys seem to affect the ability of respondents to accurately recall their medication use.1 Surveys that collect details such as the names and doses of drugs through procedures such as checking medicine cabinets or in-person review of prescription labels show high comparability with pharmacy claims data.2-8 By contrast, surveys with open-ended questions appear to be less comparable with pharmacy claims data.3-9 In one study, the sensitivity of a specific question was twice as high as an open-ended question (88% versus 41%).3 Also, claims and survey data agree more strongly for medications used regularly, such as medicines for the cardiovascular system and for diabetes, than for those used on as-needed basis, such as proton pump inhibitors.10 The literature suggests that differences in survey questions, classes of drugs, and sample populations affect the level of comparability between claims and survey data.

To date no study has examined the comparability of survey data and prescription drug claims in Canada. This study compares two sources of information about prescription drug use by people aged 65 or older in Ontario—the Canadian Community Health Survey (CCHS) and the drug claims database of the Ontario Drug Benefit (ODB) Program. The analysis pertains to cardiovascular and diabetes drugs because they are commonly used, and almost all are prescribed on a regular basis. A secondary objective is to examine the comparability of data about the use of these medications based on different questions in the 2001 and 2005 CCHS. In 2001, the questions were asked of all respondents, while in 2005, the questions were asked only of those who reported being diagnosed with the relevant conditions. Finally, individual-level factors associated with higher levels of agreement between the two data sources are examined.

Data and methods

Data sources

The data are from the drug claims database of the ODB program and the CCHS, which were linked through survey respondents' health insurance numbers. The ODB program is part of the Ontario Public Drug Programs, which collectively fund about half the total cost of prescription medications in Ontario.

This analysis concerns seniors (aged 65 or older) living in private dwellings, because the ODB program is the primary payer for this population for all prescription medicines included in the provincial formulary, and the sample is representative of this population (people younger than age 65 may be covered by private insurance plans or by the ODB program if they are eligible for social assistance). Seniors are automatically enrolled in the general ODB program, which entails an annual $100 deductible and $6.11 co-payment per dispensed drug. People whose annual income is low (less than $16,018 for single individuals; less than $24,175 for couples) can apply for reduced cost-sharing.

The ODB database records the drug name, dosage form and strength, date, quantity, and duration of the dispensation as submitted by pharmacists. An audit of 50 pharmacies in Southern Ontario found extremely high reliability of the coding of drug type, date, quantity, and duration of the dispensed drugs in the ODB claims database.11

The CCHS is a cross-sectional survey conducted by Statistics Canada, which targets the population aged 12 or older living in private dwellings. The survey excludes full-time members of the Canadian Forces and residents of Indian Reserves, Crown lands, institutions and some remote regions. This analysis drew on the Ontario component of two cycles of the survey that included medication questions: the first (2001) and third (2005) cycles.

The CCHS has optional content modules. Each module is assigned a point-value based on the average length of time needed to respond to it; health regions can select any combination of modules as long as the points do not exceed a certain threshold (32). In 2001, an optional module on medication use was administered in 29 out of 37 health regions in Ontario; in 2005, the questions on medication use were mandatory for all health regions. A previous study found no substantive difference in socio-economic, health and demographic characteristics between those who answered the optional drug module and those who did not.12 Moreover, there is no possibility of individual selection effects, because the decision to include the module was made at the health region level.

This project was approved by the Research Ethics Board of Sunnybrook Health Sciences Centre.

Sample selection

Reported and dispensed drugs were compared among CCHS respondents who had at least one drug dispensed in the 100 days before their interview (in 2001 and 2005). One hundred days is the most common prescription duration, as well as the longest duration, for the drugs examined in this study.

For both years, the sample selected for analysis consisted of respondents aged 66 or older at the time of their interview, who agreed to have their CCHS data linked to administrative data, and who had at least one prescription drug claim in the 100-day period before the interview date. The full CCHS sample for Ontario (n=37,681) and the CCHS sample for Ontario who agreed to have their data linked (n=32,848 or 87%) did not differ with regard to socio-demographic characteristics (percentage born in Canada, female, married, and with postsecondary graduation) or health status (percentage with self-assessed good health, activity limitations, reporting a visit to a physician in the past year, and having a regular medical doctor).

Sensitivity analyses were conducted using a 30-day and a 130-day period before the interview. Although the survey question asked about drug use in the past 30 days, it is expected that a longer time frame is needed to capture medications recorded in the claims data that were used in the past 30 days, but had been prescribed earlier. The 130-day window was selected to include individuals who consumed a medication 30 days before the interview, but had filled the prescription for it 100 days before consumption.

The medication questions in the 2001 and 2005 CCHS differed. In 2001, all respondents were asked a series of questions about their use of medications in multiple categories, including: "Now I'd like to ask a few questions about your use of medications, both prescription and over-the-counter. In the past month, did you take . . . (medicine for blood pressure, pills to control diabetes)?" In 2005, respondents were asked if they had any of a list of conditions (that included high blood pressure and diabetes) diagnosed by a health professional. These questions were followed by yes/no questions about medication use: "In the past month, have you taken any medicine for high blood pressure" and "In the past month, did you take pills to control your blood sugar?" Appendix Table A contains the drug identification numbers (DINs) for the drugs included in each drug class.

Table A Drug names and drug identification numbers (DINs) of anti-hypertensive medications and oral diabetes medicationsTable A Drug names and drug identification numbers (DINs) of anti-hypertensive medications and oral diabetes medications


With SAS 9.2, the prevalence of antihypertensive and oral diabetes medication use in the CCHS and ODB database was compared for the two time periods. The number and percentage of CCHS respondents who reported using blood pressure medication and oral diabetes medication among those who had a claim in the previous 100-day period (the sensitivity of the self-reported measure) was calculated, as were the number and percentage of respondents who did not report using the medication and did not have a relevant ODB claim (the specificity of the self-reported measure). With bootstrapping methodology provided by Statistics Canada,13 kappa statistics of agreement between the two data sources were calculated, along with 95% confidence intervals. Following Altman,14 kappa was interpreted as: poor (less than 0.20), fair (0.20 to 0.40), moderate (0.41 to 0.60), good (0.61 to 0.80), or very good (0.81 to 1.00).

To examine factors associated with agreement between the two data sources, logistic regression was used to model the odds of agreement, combining both sensitivity and specificity. In other words, "agreement" includes both those who reported taking the drug and had a claim in the 100 days before their interview, as well as those who did not report taking the drug and did not have a claim in the 100 days before their interview. Separate models were run for antihypertensive and oral diabetes medication use for the two survey years. Independent variables were selected based on studies that compared the reporting of medication and health care use with administrative data.3,6,15,16 These variables are age, sex, and health-related and socio-economic characteristics. Three age groups were defined: 66 to 74, 75 to 84, and 85 or older. Health status was measured by general self-assessed health (poor/fair versus good/very good/excellent). Socio-economic status was measured with an indicator of enrolment in the drug program for low-income seniors, and by highest educational attainment (at least some postsecondary versus less than some postsecondary). Survey sampling weights were used to account for the complex sampling design of the survey.


The prevalence of antihypertensive medication use was 40% in 2001 based on both self-report and pharmacy claims, and in 2005, the prevalence of use was 52% based on self-report and 49% based on claims data (Table 1). The prevalence of oral diabetes medication use was similar between the two data sources.

Table 1 Agreement between drug claims data and self-reported use of antihypertensive medications and oral diabetes medications, by period in which medication was dispensed, household population aged 65 or older, Ontario, 2001 and 2005Table 1 Agreement between drug claims data and self-reported use of antihypertensive medications and oral diabetes medications, by period in which medication was dispensed, household population aged 65 or older, Ontario, 2001 and 2005

The sensitivity of reported oral diabetes medications was higher than for reported antihypertensive medications. The sensitivity of reported antihypertensive use was slightly higher based on the targeted 2005 question than on the open-ended 2001 question. Specificity was also much higher for oral diabetes medications than for antihypertensive medications. There was little difference in specificity between the survey years.

Based on the kappa statistics, agreement between the data sources for oral diabetes medications was good and very good in 2001 and 2005, respectively. Agreement for antihypertensive medications was moderate. Implementation of targeted questioning in 2005 appeared to be associated with improved agreement for both drug categories.

Sensitivity analyses using a 30-day and a 130-day window to measure claims data show that the results are sensitive to the length of the window (Table 1). Not surprisingly, the prevalence of medication use, as well as sensitivity and overall agreement, were significantly reduced with the 30-day window. The results remained largely unchanged using the 130-day window.

Overall agreement between data sources, defined as reporting use of the drug and having a corresponding pharmacy claim, or not reporting use of the drug and not having a corresponding pharmacy claim, was near perfect for oral diabetes medications (97% in both 2001 and 2005). For antihypertensive medications, overall agreement was lower: 75% in 2001 and 78% in 2005.

Logistic regression was used to model the individual-level factors associated with overall agreement for antihypertensive medications (Table 2). The analyses revealed that the only statistically significant associations were with age (older individuals were less likely than those aged 66 to 74 to have agreement between the data sources) and health (those with poorer health had lower levels of agreement between the data sources).

Table 2 Adjusted odds ratios relating selected characteristics to agreement between drug claims data and self-reported use of antihypertensive medications, household population aged 65 or older, Ontario, 2001 and 2005Table 2 Adjusted odds ratios relating selected characteristics to agreement between drug claims data and self-reported use of antihypertensive medications, household population aged 65 or older, Ontario, 2001 and 2005


This is the first study to assess agreement between a national health survey (the CCHS) and pharmacy claims data. Agreement between the two data sources was high for oral diabetes medications, but moderate for antihypertensive medications. The prevalence of medication use was comparable for both drug classes.

The way in which the CCHS asked questions about medication use differed between the two survey cycles. The more targeted 2005 approach improved agreement with claims data for both drug classes. A 100-day time period for measuring claims data appears to have been adequate to capture medicines consumed in the previous 30 days.

In multivariate analysis, agreement between self-reported and claims data for antihypertensive medications was higher for younger than for older seniors, and for those in better health compared with those reporting poor/fair general health. Another study, too, found lower odds of agreement between self-reported and administrative data on health care utilization among older individuals.16

The higher level of agreement between the data sources for oral diabetes medications than for antihypertensive medications has been reported elsewhere.6 It is possible that some people may not be aware that they have hypertension,17-20 and therefore, are not cognizant of the type of medication they are taking. As well, the CCHS asked respondents about medications for "blood pressure," but it is possible that patients may be taking antihypertensives for other reasons (for example, post-myocardial infarction or heart failure), and so do not report it to the CCHS.


A number of difficulties arise in comparing different sources of prescription drug use data. Surveys measure drugs that are actually consumed by the patient, whereas pharmacy claims measure drugs that are dispensed. After it has been dispensed, a drug prescribed for a chronic condition may not be consumed if the patient does not adhere to the treatment plan.21-23 The patient may forget to take the drug, or start taking the drug but discontinue use because the symptoms are reduced or relieved or because of adverse effects.24,25 Therefore, the comparability of self-reported medication use and pharmacy claims data is complicated by the inability to determine if inaccurate reporting stems from recall problems about the types of medications taken26 or from non-adherence. Levels of non-adherence are likely to be greater for conditions that are asymptomatic such as hypertension. Since a binary use/no use variable was employed, this study includes people with imperfect adherence, but not those who did not take the medication at all in the 100-day period. Another reason for discrepancies between the two data sources is that individuals may report complementary therapies that they used for hypertension as "high blood pressure medications."27

The pharmacy claims data are missing information on individuals who purchased a medicine that is not in the ODB formulary. However, the majority of medications available for the classes of drugs investigated in this study were included in the ODB formulary, so missing data because of private purchase are likely to be minimal.28


The results of this analysis suggest that self-reported medication use is an accurate and valid data source for measuring drug exposure among the elderly for medications taken on a chronic basis. Accuracy appears to be improved with a more targeted rather than an open-ended approach to asking questions about medication use. In the case of antihypertensive medications, researchers should consider possible underreporting, particularly among people older than 75 and those in poor health. The availability of linked data offers a unique opportunity to estimate the comparability of these data sources, and to conduct future research on patterns of medication use.


The authors acknowledge the support of The Canadian Institutes for Health Research, the Ontario Ministry of Health and Long-Term Care, the Canadian Health Services Research Foundation and the Lupina Foundation. Dr. Law receives salary support through a New Investigator Award from the Canadian Institutes of Health Research and an Early Career Scholar Award from the Peter Wall Institute for Advanced Studies. Dr. Bayoumi is supported by a Canadian Institutes of Health Research/Ontario Ministry of Health and Long-Term Care Applied Chair in Health Services and Policy Research. This study was, in part, supported by an operating grant from the Canadian Institutes of Health Research (MOP-221233, "For Whom the Bill Tolls: Private Drug Insurance in Canada," P.I. Michael Law). The authors thank the participants of McMaster University's Centre for Health Economics and Policy Analysis polinomics seminar for their comments on earlier versions of this paper. The authors also thank Michael Paterson, Michael Manno and Anjie Huang for their help with the statistical analysis. The Centre for Research on Inner City Health is supported in part by a grant from the Ontario Ministry of Health and Long-Term Care. The views expressed in this article are those of the authors, and no official endorsement by supporting agencies is intended or should be inferred.