Data quality, concepts and methodology: Data quality, concepts and methodology

Warning View the most recent version.

Archived Content

Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please "contact us" to request a format other than those available.

Estimation of research and development expenditures in the higher education sector, 2008/2009

Definitions

Natural sciences and engineering

The natural sciences and engineering (NSE) field embraces the disciplines of study concerned with understanding, exploring, developing or utilizing the natural world. Included are the engineering, mathematical, life and physical sciences.

Social sciences and humanities

The social sciences and humanities (SSH) field embraces all disciplines involved in studying human actions and conditions and the social, economic and institutional mechanisms affecting humans. Included are such disciplines as anthropology, demography, economics, geography, history, languages, literature and linguistics, law, library science, philosophy, political science, psychology, religious studies, social work, sociology, and urban and regional studies.

Scientific research and experimental development (R&D)

Creative work undertaken on a systematic basis in order to increase the stock of scientific and technical knowledge and to use this knowledge in new applications.

The central characteristic of R&D is an appreciable element of novelty and of uncertainty. New knowledge, products or processes are sought. The work is normally performed by, or under the supervision of, persons with postgraduate degrees in the natural sciences or engineering.

An R&D project generally has three characteristics:

  1. a substantial element of uncertainty, novelty and innovation;
  2. a well-defined project design;
  3. a report on the procedures and results of the projects.

Canadian business enterprises

This sector is composed of business and government enterprises, including public utilities and government owned firms and frequently referred to as the industry sector. Incorporated consultants providing scientific and engineering services are also included. Industrial research institutes located at Canadian universities are considered to be in the university sector.

Higher education

The higher education sector is composed of all universities, colleges of technology and other institute of post-secondary education, whatever their source of finance or legal status. It also includes all research institutes, experimental stations and clinics operating under the direct control of, or administered by, or associated with, the higher education establishments.

Canadian private non-profit institutions

Charitable foundations, voluntary health organizations, scientific and professional societies, and other organizations not established to earn profits comprise this sector. Private non-profit institutions primarily serving or controlled by another sector should be included in that sector (e.g., the Pulp and Paper Research Institute is in Canadian business enterprises).

Canadian provincial and municipal governments

Departments and agencies of these governments form this sector. Government enterprises, such as provincial utilities are included in the Canadian business enterprises sector, and hospitals in the Canadian non-profit institutions or university sector.

Other Canadian performers

This sector includes all individuals or organizations not belonging to any of the above sectors. In particular, it includes provincial research councils and foundations.

Foreign performers

All foreign governments, foreign companies (including foreign subsidiaries of Canadian firms), international organizations, non resident foreign nationals and Canadians studying or teaching abroad, are included in this sector.

Methodology of estimating higher education research and development expenditures (HERD)

1. Introduction

Research is an integral part of higher education institutions' mission. Faculty do research as part of their teaching function. They also perform research sponsored by other sectors of the economy. Total research and development performed by the higher education sector is the sum of expenditures made from funds received from other organizations (sponsored research) and the monies spent from the institutions' own budgets (non-sponsored research).

Higher education is not a sector in the System of National Accounts, but in the system of research and development, gross domestic expenditures on research and development (GERD), it is separated because of its critical role in the creation and dissemination of new knowledge. The Organisation for Economic Cooperation and Development (OECD) describes it as "all universities, colleges of technology and other institutes of post-secondary education, whatever their source of finance and or legal status. It also includes all research institutes, experimental stations, and clinics operating under the direct control of, or administered by, or associated with, the higher education establishments." 1 

Estimation of HERD can be approached in two ways: sources of funds (income) approach and research performed (expenditure) approach. However, they yield different results as all the funds received by institutions in any one year may not be spent in that same year.

Statistics Canada employs a combination of the two approaches due to data constraints. The expenditure approach is used to estimate total HERD while details -- sources of funds and science fields -- are based on the income approach. Any discrepancies in estimates derived from the two different approaches are fully resolved to ensure all the data presented in this release are consistent.

As mentioned above, higher education sector R&D has two main components: sponsored and non-sponsored. Each of these is further sub-divided into direct and indirect costs:

  1. Direct sponsored research is the university research funded by external organizations in government, business, private not-for-profit, and the foreign sectors. Direct cost refers to expenditures that can be easily and accurately attributed to a single project such as researchers' salaries;
  2. Direct non-sponsored research is a co-product of teaching. It is an integral part of the teaching function; and
  3. Indirect cost of sponsored and non-sponsored research. This is the cost of research that cannot be easily and accurately traced to a single project or activity because it is jointly incurred by numerous research projects and activities going on in an institution at the same time and therefore must be apportioned to each project according to its usage of the institution's facilities and services. Examples include fire insurance on a building, utility bills and the use of central services.

2. Methodology

The current estimation methodology was proposed by the 1999 Working Group and has been applied since 1998/99. The principal source of data is the annual survey, Financial Information of Universities and Colleges, conducted by the Canadian Association of University Business Officers (CAUBO). Tables from this survey are provided by the Tourism and Centre for Education Statistics Division of Statistics Canada.

R&D Expenditure (expenditure approach)

Total HERD is the sum of direct sponsored research, direct non-sponsored research and indirect cost of sponsored and non sponsored research. In the estimation model, an additional module is added to account for affiliated hospitals not included in these components.

  1. 1. Direct sponsored research

Direct sponsored research expenditure is derived from data in CAUBO Table 22  As the source does not separate direct and indirect costs, 95% of the sponsored research expenditure reported to CAUBO is assumed to represent direct sponsored research; the remaining 5% is assigned to indirect cost representing recoveries made from the sponsors.

  1. 2. Direct non-sponsored research

Faculty divide their time among the three primary functions; teaching, research and community services. The time spent on research when it is undertaken as part of the teaching function is defined as non-sponsored research. Central to the estimate of the value of direct non-sponsored research are the portion of faculty time spent on this type of research and faculty salaries.

In order to estimate the amount of faculty time spent on research, Statistics Canada commissioned a faculty time use survey in 2001. Academic staff at Canadian universities was the target population. 3  After analysis of the results and some modifications, faculty research time coefficients were derived, detailed by eight academic disciplines and three university sizes. They are summarized in Table A. Note that coefficients were estimated for total, not non-sponsored research, but an adjustment was made to faculty salaries -- which can be more easily divided between sponsored and non-sponsored research – to avoid double counting.

These coefficients are applied against the number of full-time faculty in each of the eight teaching disciplines and the salaries of academic ranks reported by CAUBO for each institution. It is further assumed that all faculty members are at the same salary levels in the absence of more detailed salary information from existing sources. The resulting figure is reduced by the amount of salaries funded by the sponsors.

Size classification of universities is based on three criteria (Table B): the amount of expenditure on sponsored research (reported by CAUBO); the proportion of sponsored R&D expenditure to general operating expenditure; and finally, the number of doctoral programs offered by the institution. An institution has to satisfy two of the three conditions to decide its group. However, if it is judged to be small on two criteria and large on the third it is assigned to the medium group.

It should be noted that the final objective is not to create an individual ranking for universities but rather to group them into three size groups to make possible R&D expenditure estimates at the aggregate level.

  1. 3. Indirect cost of sponsored and non-sponsored research

In the estimation model, indirect costs are the sum of four components:

  1. federal government indirect cost payments - it is taken from CAUBO Table 6;
  2. indirect costs recovered from non-federal sponsors - it is embedded in CAUBO's data, and assumed to be 5 per cent of the sponsored research expenditure;
  3. indirect cost not reimbursed by sponsors – it is derived as a fraction of direct sponsored research; it is discussed in detail below; and finally
  4. indirect cost of non-sponsored research – it is estimated the same way as the indirect cost of sponsored research not reimbursed by sponsors.

As indicated, data for the first two components are available, but the third and fourth items are estimated by calculating the indirect to direct university operating cost ratio. This ratio is computed in several steps described below. The methodology is a short-cut version of the very detailed method employed in the 1982 CAUBO study. 4 

A. Total operating cost is defined as the sum of expenditures from three funds -- general operating; special purpose and trust; and sponsored research; the other funds that higher education institutions maintain – capital, ancillary and endowment -- are assumed to contain no operating cost.

B. Second, indirect cost portion of each of the three funds is calculated. It is accomplished by calculating the indirect to direct operating cost ratio for the general operating fund for which most detail is available and applying it to special purpose and trust fund for which no detail is available.

  1. All expenditure from all items 5  in the general operating fund (except on student services and academic salaries) is assumed to represent indirect operating cost; only academic faculty salaries are apportioned, 11% to indirect cost and 89% to direct cost, 6  based on the findings of a 1982 study that 11% of faculty time was taken up by the various administrative duties that support teaching and research;
  2. As an independent ratio cannot be calculated for student services and for special purpose and trust fund because of the lack of detailed data, they are assumed to contain direct and indirect costs in the same proportion as the general operating fund;
  3. Five percent of the sponsored research fund is assumed to represent indirect operating cost;
  4. Thus total indirect cost is the sum of the three items, Ba to Bc;

C. Third, direct operating cost is derived residually by subtracting indirect operating cost (Bd) from the total operating cost (A).

D. Finally, dividing indirect operating cost (Bd) by direct operating cost (C) we obtain the indirect university operating cost ratio. These estimates are made, one each for small, medium and large institutions, using the classification criteria listed in Table B above.

These ratios are applied to direct sponsored research expenditures and direct non-sponsored research expenditures to arrive at an estimate of indirect cost of research not reimbursed by sponsors and indirect cost of non-sponsored research.

  1. 4. Teaching hospitals not included elsewhere

Data sources are constantly reviewed to ensure full coverage of teaching hospitals. Statistics from the Survey of Hospitals collected by The Canadian Institute for Health Information are used to calculate the direct and indirect cost of research performed by teaching hospitals not included elsewhere.

  1. 5. Total HERD

Total HERD is then the sum of (1) direct sponsored research expenditures, (2) direct non-sponsored research expenditures, (3) indirect cost of sponsored and non-sponsored research, and (4) direct and indirect cost of research at teaching hospitals not covered elsewhere.

Sources of funds, income approach

Sources of funds data obtained from CAUBO Table 1 require two main refinements before they can be used; reconciliation of sector definitions and resolving discrepancies between income and expenditure data.

First, the CAUBO sector definitions do not conform to those used in the higher education sector R&D. There is a good mapping for federal government, provincial governments and the foreign sectors but, as shown in Table C, business and not-for-profit sectors had to be constructed from various components. Furthermore, certain items, including tuition and other fees, sales of goods and services and other investment, are not related to research and were excluded.

Second, income and expenditure sides of sponsored research fund need to be reconciled. This is first done at the aggregate level for each higher education institution because detail is only available for the income side. When income is higher than expenditure it is adjusted down to the level of expenditure and the difference is prorated to the sources; however, no adjustment is made when expenditure exceeds income.

Expenditure by field of science, income approach

Estimates of research expenditure by science type are based on adjusted income, described in the preceding section. As summarized in Table D, allocation is funding institution-specific and takes into account organization's mandate and statistical information, wherever available.

3. Historical continuity of data series

This working paper contains fifteen years of HERD estimates based on the revised estimation procedure first used for 1998/1999 estimates. During the 1999/2000 estimation procedure, revised faculty time coefficients on research were used. These new coefficients were then applied to the 1998/1999 estimates as we feel those new coefficients better reflect university research activity levels. During the 2000/2001 estimation procedure a better analysis of "teaching hospitals" expenditures was completed. Some overlap between the private non-profit sector and the higher education sector was discovered and resolved. One will see historical revisions in both sectors back as far as 1991/1992.

During the 2001/2002 estimation procedure the one-time grant awarded to the universities for indirect costs related to sponsored research was reviewed. The estimation system had to be modified to ensure those costs were firstly sourced to the federal government and secondly were not double counted. In 2003/2004 the indirect costs grant awarded from the federal government became an annual payment to universities. The estimation system ensures these payments are not counted twice.

The inclusion in CAUBO statistics sponsored research activities performed in teaching hospitals is constantly reviewed as some teaching hospitals are included and some are not. For those that are not included, estimates are made using information from the Annual Hospital Survey, collected by The Canadian Institute for Health Information (CIHI).

4. Future work

Statistics Canada held a technical workshop to review the assumptions and methodology of estimating HERD, in October 2009, as part of its regular review of data quality. The Workshop made several recommendation concerning, among others, data sources, assumptions, methods, sector coverage, and the impact of new developments, particularly the Networks of Centres of Excellence and the Knowledge Infrastructure Program. 7  They also included the formation of a Working Group to make proposals for implementing the recommendations. The Working Group was formed in December 2009 and has been holding discussions to prepare its report.