The Consumer Price Index and the CPI Enhancement Initiative 2008-2009 to 2012-2013

Evaluation Report

May 2013

  • Executive summary
  • 1. Introduction
  • 2. The Consumer Price Index
    • 2.1 Background and context
    • 2.2 Objectives and outcomes
    • 2.3 Program management, governance and key stakeholders
  • 3. Evaluation methodology
    • 3.1 Evaluation scope, timing and quality control
    • 3.2 Evaluation questions by issue area
    • 3.3 Methods
  • 4. Findings — relevance
    • 4.1 Core issue 1: continued need for the program
    • 4.2 Core issue 2: alignment with government priorities
    • 4.3 Core issue 3: consistency with federal roles and responsibilities
  • 5. Findings — design and implementation of the CPI-EI
    • 5.1 CPI-EI design decisions
    • 5.2 CPI-EI implementation
    • 5.3 CPI-EI governance
    • 5.4 Implementation challenges
  • 6. Findings — performance
    • 6.1 Core issue 4: achievement of expected outcomes
    • 6.2 CPI information available in public domains
    • 6.3 Stakeholder engagement
    • 6.4 Core issue 5: demonstration of efficiency and economy
  • 7. Conclusion and recommendations
    • 7.1 Conclusion
    • 7.2 Recommendations
  • 8. Management response and action plan
  • Appendices
    • Appendix A — CPI and CPI-EI logic models
    • Appendix B — Governance and project management structure
    • Appendix C — Detailed evaluation matrix
    • Appendix D — Summary of key activities and outputs of CPI-EI
    • Appendix E — Examples of relevant management practices
    • Appendix F — IMF data quality assessment framework, Canada
    • Appendix G — Literature and documents reviewed

Executive summary

This report presents the findings and recommendations from the 2012/2013 evaluation of the Consumer Price Index (CPI), including the CPI Enhancement Initiative (CPI-EI).

The evaluation was designed and conducted in accordance with Treasury Board (TB) Policies and Guidelines as outlined in the TB's Policy on Evaluation (April 2009). The evaluation objective is to provide credible and neutral information on the ongoing relevance and performance of the CPI. The evaluation focuses on program performance from 2008/2009 to 2012/2013.

Overview of the CPI

The CPI, produced by Consumer Prices Division (CPD), is a key Statistics Canada product. The CPI is a targeted measure used by key stakeholders to assess changes in consumer prices over time and to determine inflation rates.

In 2009/2010, the Government of Canada introduced a five-year, $45-million initiative to enhance the CPI, based on economic theory and best practices, to estimate price changes more accurately and to use its available resources more cost effectively. The initiative is focused on changes to the sample size and design, as well as basket updates and quality adjustments, in order to address known biases that are a normal characteristic of fixed basket CPIs such as commodity substitution and representative estimates. These changes are supported by IT infrastructure projects, which are critical enablers to the CPI-EI, and by an HR strategy.

Evaluation methodology

This evaluation was led by the Evaluation and Professional Practices Division with the support of Intergage Consulting Group Inc., an Ottawa-based consulting firm. Field work was conducted from August 2012 to March 2013. Data for this evaluation were collected from four lines of evidence: a literature review, a document review, a jurisdictional review, and interviews with key informants. Logical and statistical analyses were used. Findings were correlated to identify trends and patterns.

This evaluation has these limitations:

  • program size and complexity in relation to the evaluation budget
  • heavy focus on the enhancement initiative
  • high diversity of key informant interview groups
  • inherent biases of key stakeholders.

Findings

Relevance

The CPI is an important measurement used by the Government of Canada and other key stakeholders to inform economic and monetary policy. It is used in the development of the GDP measure. The Bank of Canada uses the CPI to measure rates of inflation; Finance Canada uses it to help determine transfer payments to the provinces and territories. Internationally, the CPI is one of the most widely-used statistics for measuring price changes and economic performance; it is the most visible and credible measure to anchor inflation expectations. Consequently, there is a continued need for the CPI; it remains highly relevant.

The CPI is aligned with federal and provincial government priorities, and is used by all levels of government. The production of the CPI is an ongoing priority for Statistics Canada. The Government of Canada has a legitimate role and responsibility to produce the CPI. Statistics Canada, given its ability to provide objective and non-partisan statistics, should be the agency to do so.

Design and Implementation of the CPI-EI

The evaluation found evidence that the CPI-EI was necessary to address known weaknesses of the CPI, and to ensure that the CPI continues to be a high-quality, reliable index. Evidence suggests that the design of the CPI-EI is sound, and that the initiatives underway are the right ones to improve the CPI. Furthermore, there is evidence that the CPI-EI is being implemented as planned; some elements are being implemented ahead of schedule.

The evaluation found CPI-EI governance arrangements to be appropriate and sufficient to support the initiative and the ongoing management of associated risks. However, there is some concern about the clarity of roles and responsibilities within CPD, and about governance arrangements and mechanisms understanding for managing cross-cutting issues at lower levels in the division. Another concern is the potential erosion of subject-matter expertise — either through turnover or retirement — and the potential risks and impacts on the CPI.

Achieving Expected Outcomes

Good progress has been made toward realizing CPI and CPI-EI outcomes at the immediate and intermediate level. CPI is timely, relevant and, despite some weaknesses in availability of information and limitations of the index, credible. The CPI is an excellent index when used for its intended purposes. The evaluation's findings confirm that, as a result of the CPI-EI, biases in the CPI will be reduced, leading to a more reliable and credible price index with greater internal capacity. As for the ultimate outcome, that "Canadians have accurate, timely, relevant set of consumer price indexes meant to support a variety of key policy programs," the CPI program is well positioned. More specifically:

Data production

Canada's CPI meets international standards and is an index that is trusted, credible, relevant and timely. Evidence also revealed that the changes ushered in under the CPI-EI are consistent with the approach found in other countries and that the CPI-EI is needed to maintain continued and possibly increase the overall confidence in the CPI.

Dissemination and communication

Statistics Canada's CPI data, research, publications and methodology are made available to the public in an open manner via Statistics Canada's website. However, at times, stakeholders have had difficulty finding information and a Statistics Canada contact person to respond to questions about the CPI.

CPD appears to have well established relations with their primary clients, but has invested less time in engaging a broader range of stakeholders in the CPI and CPI-EI. Stakeholders desire improved communications and dissemination of more detailed information and data, as well as more frequent updates.

Research and development

There is evidence that the CPI-EI is well positioned to achieve its intermediate outcome and the CPI data and index withstand public and expert scrutiny. However, the evaluation found a need for continual research and development to ensure that the CPI remains highly relevant and credible.

Information technology

The IT enablers introduced to support the CPI-EI are meeting CPI requirements and are improving CPI data collection methods. The CPI-EI has led to the enhancement of the IT infrastructure, by introducing improvements, greater efficiencies and modernization of CPI computer systems and tools. However, there is a potential risk that future resource constraints may limit CPI's IT evergreen strategy to renew key IT systems over rolling five-year cycles.

Economy and Efficiency

A review of program financial data indicates that, from a financial perspective, the CPI-EI project is being implemented according to plan. Expenditures on the CPI and the CPI-EI appear to be in accordance with the original budget. The CPI seems to use an efficiency-based approach in its practices. Some evidence shows that the CPI has efficient administrative and operational procedures.

There is room to improve efficiency by implementing a performance measurement system, including harmonizing processes used in the CPI with those used in the System of National Accounts and the Survey of Household Spending Household Survey. Some alternative approaches and considerations for the CPI were identified, such as moving the CPI to a one-year update cycle; making use of alternative data sources and data collection methods; examining the benefits of introducing a superlative index and allowing for revisions; and using seasonal adjustments.

Recommendations emerge from the evaluation findings that advocate strategic planning for sustainability after CPI-EI complete implementation, enhanced governance at the sub-project level, improved communication with stakeholders, accessibility of CPI products, and strengthened capacity for systematic performance measurement to demonstrate the achievement of results.

1. Introduction

1.1 Purpose

This report presents the findings and recommendations from the 2012/2013 evaluation of the Consumer Price Index (CPI), including the CPI Enhancement Initiative (CPI-EI). The evaluation report provides information on the program, the methodology used, the findings and conclusions for each of the evaluation questions, and overall conclusions and recommendations. This evaluation was conducted as prescribed by the Departmental Risk-based Audit and Evaluation Plan for 2012/2013 to 2016/2017, approved by the Departmental Evaluation Committee on March 27, 2012. The project was managed and carried out by the Evaluation and Professional Practices Division in Audit and Evaluation Branch.

2. The Consumer Price Index

This section briefly describes the CPI and the CPI Enhancement Initiative (CPI-EI), including the program's objectives, governance and resources.

2.1 Background and Context

Produced by Consumer Prices Division (CPD), the CPI is a key Statistics Canada product. It is the target measure used by the Bank of Canada to set interest rates. It is also used to adjust public- and private-sector arrangements and contracts such as pensions, collective agreements, rental agreements and tax brackets, so that price changes are factored into payments and transfers. In addition, the CPI is used to deflate 'nominal' values relating to consumer incomes and expenditures to produce 'real' measures, where price changes are eliminated—this helps reveal underlying changes in the economy or individual well-being. The CPI is also used for many ancillary analytical purposes, such as understanding the business cycle, making investment decisions or forecasting economic trends.

When it was first introduced in the early 1900s, the CPI was based on 29 food items, 5 fuel and lighting items, and the cost of rent in some 60 cities. Today, the CPI reflects prices for virtually every category of good and service Canadians can buy, right across the country. It is constructed by collecting some 72,000 price observations monthly, tracking about 600 goods and services in the Canadian economy and covering 1200 representative products. The CPI measures the average change in retail prices seen by all consumers in Canada. Items are weighted to account for typical spending patterns and how often consumers buy certain items (for example, appliances are replaced only rarely). The CPI is continually updated to be as comprehensive and representative as possible.

The CPI currently uses 2002 as its base year, the point in time when the index equals 100. The CPI is based on the overall cost of a fixed basket of goods and services bought by a typical consumer relative to price of the same basket in the weight reference year. Because it includes such a broad range of goods and services, the CPI is often used to estimate changes in the cost of living.

The CPI has become the primary tool for measuring inflation given the central role of these data in the management of Canada's economy. The CPI is not a dollar value, like gross domestic product (GDP): it represents an index number, a percentage of measured price change from the base year. As the prices of different goods and services do not all change at the same rate, a price index is designed to reflect their average movements.

2.1.1 The CPI-EI

In 2009/2010, the Government of Canada introduced this five-year, $45-million initiative to improve the CPI based on economic theory and best practices, in order to estimate price changes most accurately and to use the CPI's resources most cost-effectively. Statistics Canada received new funding to permit improvement to Canada's CPI in these areas:

  • Sample size activity aims to increase the number of prices collected. The number of observations taken relative to the size or characteristics of a population can affect the quality of the inferences made about the population being measured. As the sample sizes of some products in Canada's CPI are considered too small to be representative, the CPI-EI aims to increase the number of prices collected of goods and services purchased by Canadians. The objective is to double the number of prices collected (from 60,000 to 120,000) within the next five years — approximately 5,000 additional price quotes in each of the first three years, and 15,000 in the fourth year and 30,000 in the fifth.
  • Sample design activityaims toimprovethe selection of stores and products to be sampled. These changes should better reflect what Canadians are actually purchasing, as well as shifts in consumer product and store preferences. They should improve timeliness in the introduction of new outlets and products, as well as avoidance of oversampling of less significant products and/or undersampling of popular items. The CPI-EI objective is to improve the sample design by using more scientific sampling methodologies and better targeted, more efficient collection. These changes should be fully operational by year 5.
  • Basket updates activity aims to ensurethat the weights of the products in the CPI basket are updated more frequently and processed more quickly. Prior to the CPI-EI, the CPI basket was updated every four years: each update was based on data 18 months previous to the year the expenditure data were gathered. This led to the basket becoming gradually outdated and not being reflective of the recent changes in consumers' buying patterns. CPI-EI committed to providing a biannual update of CPI basket weight by 2013/2014 and to reducing the time to incorporate a new basket into the CPI to 12 months to make updates a better and earlier reflection of consumer spending.
  • Quality adjustment activity aims to develop new methodologies to better distinguish pure price movement from that resulting from changes in product quality or product characteristics. Currently, about one-quarter of products in the CPI are quality adjusted. The CPI-EI targeted the development of better methodologies for an additional 75 products by 2014/2015, bringing the total number of adjustment to 225 products. Coverage will be extended to appliances, furniture and electronics.

These areas are supported by two key IT infrastructure projects that are critical enablers to the CPI-EI.

  • Data collection infrastructure improvements aim to increase management capabilities to control collection activities in the field. This will better align collection activities to the CPI's data needs. The intention is to ensure that products and stores are better targeted and that more information is collected for improved quality assurance. Hand-held devices used in the field for data collection will be replaced and their software upgraded.
  • Data processing changes aim to improve the CPI's data processing and index estimation infrastructure. The index estimation system (MPS), which constructs indexes from the raw data, is built on technology and systems design principles that are over 20 years old. It is also a complex system not designed to efficiently accept frequent changes to basket methodology. The objective is to modernize the technology used in the index estimation and improve its documentation, resulting in a system that is easier to use and maintain.

To support these four key activities and two IT infrastructure initiatives, Statistics Canada developed a recruitment strategy to hire new employees and a dedicated training program to shorten learning curves.

In 2010/2011, Statistics Canada started to receive funding to undertake the CPI-EI. In 2012/2013, the CPI-EI began its third year of implementation. In this fiscal year, the initiative has focused on developing, piloting and implementing new business processes, IT systems and sample changes.

2.2 Objectives and Outcomes

2.2.1 The CPI

The CPI's objective is to reflect changes in consumer prices experienced by Canadians. This could be achieved through a diverse set of activities, such as collecting prices, applying stringent quality control measures to produce error-free and quality adjusted prices, providing representative weights at timely intervals, periodically re-evaluating and improving methods in line with international standards, as well as analyzing and disseminating data.

The immediate outcomes from these activities are to produce trusted data, put information products in the public domain and ensure that released data rest on a robust IT infrastructure. The intermediate outcomes include supporting key programs; providing high quality service to users; enhancing the design of the program (i.e., concepts, methods and data sources) and its IT infrastructure. The long-term outcomes are to ensure that Canadians have access to accurate, timely and relevant set of consumer price indexes meant to support a variety of key policy programs (monetary, fiscal and social), as well as meeting System of National Accounts (SNA) requirements.

2.2.2 The CPI-EI

The CPI-EI's objective is to produce a better measure of price movement in Canada based on sound and internationally respected methodologies and supported by a more efficient and stable technical infrastructure.

Through the four key activities and two support initiatives mentioned in the section above, the anticipated CPI outcomes include: more accurate (representative) product prices (averages); better reflection of what Canadians are buying, leading to a more representative CPI; better understanding of changing habits of Canadians; and better estimates of pure price change (removing noise in the movement of prices arising from changes in product characteristics). Like the CPI, the CPI-EI supports the long-term outcome: "improved quality of policy decision making from using an enhanced CPI."

The enhancement initiative is embedded in the CPI. CPI-EI's activities are meant to improve the CPI's ability to achieve stated outcomes. For further details, Appendix A presents logic models related to the CPI- and the CPI-EI.

2.3 Program Management, Governance and Key Stakeholders

2.3.1 Program Management

CPD, a member of the Business and Trade Statistics Field at Statistics Canada, is responsible for the overall management of the CPI and the CPI-EI. The division's mandate is to produce the CPI based on timely and relevant data and analysis of consumer price change temporally (over time) or spatially (geographically).

CPD comprises the production sub-division and the development sub-division, which are involved in the CPI and CPI-EI.

Production sub-division is responsible for producing a wide range of temporal (base funded) and spatial (cost recovery funded) price indexes monthly. It is also responsible for coordinating releases, responding to information requests and developing and integrating a divisional information management framework.

Development sub-division is chiefly concerned with developing methodologically and conceptually sound processes to ensure ongoing and sustainable CPI relevance and quality. It also produces purchasing power parities and spatial program outputs. This work is conducted as part of sectional operations and projects that are part of the CPI-EI. This sub-division hosts the Cygnus development section. This group is responsible for redesigning the index estimation system, and for building subject-matter expertise on price index estimation.

2.3.2 Governance and Project Management Structure

A number of organizational changes were made to the CPI to support the introduction of the CPI-EI and the Departmental Project Management Framework Guidelines in April, 2012. CPD distinguishes the governance structure from the project management structure. The CPI-EI governance consists of two tiers of steering committees (decision-making bodies): a director general-level steering committee for the CPI-EI, responsible for oversight of the whole CPI-EI; and three director-level steering committees, chaired by the director of CPD, which oversee components of the CPI-EI.

The project management structure consists of boards and committees that are responsible for the strategic aspect of the program, including the management board, the CPI-EI Management group and the sub-project groups.

More details on governance and the CPI structure are presented in the project governance organization structure chart in Appendix B.

2.3.3 CPI Key Stakeholders

CPD has several internal and external key stakeholders that provide inputs, such as raw data or services, or who benefit from or use the outputs, such as price indexes.

Internal stakeholders

Data providers

Within Statistics Canada, the Consumer Prices Program has a close relationship with several divisions that collect/provide price or expenditure data. Collection Planning and Management Division (CPMD) manages the collection infrastructure and the interviewers who gather much of the consumer price data in the field.

The other major group of data suppliers is from Statistics Canada's Social Statistics Field. Income Statistics Division is the primary data provider for the construction of the CPI basket's weights; Labour Statistics Division collects information on residential rents on behalf of CPD; Income Statistics Division collects information for the CPI in the Survey of Household Spending; and the Centre for Education Statistics collects data on tuition fees.

The third data supplier group is Producer Prices Division (PPD). It provides not only raw data, such as motor vehicle discounts, but also indexes such as the New Housing Price Index and the Computer Price Index, which are used by CPD in calculating the CPI.

Data users

CPD's main internal client is the SNA, which uses the CPI in constructing data series for personal expenditures at constant prices. SNA also uses CPI prices for a basket of goods and services specified by the Organization for Economic Co-operation and Development (OECD) for the construction of Purchasing Power Parities.

External stakeholders

CPD works with a wide variety of external stakeholders that comprise partners, data users and the Price Measurement Advisory Committee (PMAC). The PMAC provides advice on concepts, methods, and priorities to CPD and PPD on all aspects of price measurement. The committee meets annually and each major Prices Program component is covered at a PMAC meeting over time, about every five years.

External and Internal Partners that Provide Input to CPD

The international community, represented by the Ottawa Group, the International Working Group on Price Statistics and the joint United Nations Economic Commission for Europe–International Labour Organization meeting on consumer price indexes – is key to contributing to the advancement of price measurement and learning from other countries' experiences with their national CPI programs.

Provincial and territorial representatives, or 'focal points', meet under the auspices of the Secretariat of Federal-Provincial-Territorial Relations. At their meetings, they ask questions about the concepts, methods and data used in the CPI and provide feedback on the provinces' satisfaction as users of the program.

Data users

CPI data are used by a wide range of external stakeholders, such as

  • Canadians, since the purchasing power of money is affected by changes in prices, the CPI is useful to virtually all Canadians
  • federal departments and entities including the Bank of Canada, Finance Canada, the Canada Revenue Agency and social policy departments such as Human Resources and Skills Development Canada
  • provincial and territorial governments
  • the private sector, for indexing contracts and pensions
  • national associations that represent the interests of various stakeholder groups
  • other such as the International Monetary Fund, the Bureau of Labour Statistics and the Organization for Economic Co-operation and Development.

Figure 1. Consumer Prices Division and its stakeholders

 Divisions of Consumer Prices and placement of stakeholders

Notes
BSMD Business Survey Methods Division
CTCES Culture, Tourism and the Centre for Education Statistics
CPMD Collection, Planning and Management Division
HRSDC Human Resources and Skills Development Canada
ISD Income Statistics Division
LSD Labour Statistics Division
LHSAD Labour and Household Surveys Analysis Division
PMAC Price Measurement Advisory Committee
PPD Producer Prices Division
SNA System of National Accounts
Source: Adapted from Consumer Prices Division's 2010 Quadrennial Program Review

Description for Figure 1. Consumer Prices Division and its stakeholders

2.3.4 CPI Program Resources

While Statistics Canada receives ongoing funding for the Consumer Price Index, since fiscal year 2010/2011 the CPI program has also received supplemental A-base funding to implement the CPI-EI for a total of $17,238,157 in 2012/2013 (Table 1). According to internal data, this new funding was allocated to all activities related to the CPI to support new activities. Total funding for the CPI and the CPI-EI is detailed in Table 2.

Table 1. Summary of New A-base Funding – Budgeted Expenditures, 2010/2011 to 2012/2013
New funding for CPI-E Fiscal year
2010/2011 2011/2012 2012/2013 Total
Vote 105, program expenditures ($)
Personnel 2,550,967 4,369,115 6,975,278 13,895,360
O&M 901,213 890,078 2,052,881 3,844,172
Subtotal 3,452,180 5,259,193 9,028,159 17,739,532
EBP @ 20% 510,194 873,823 1,395,055 2,779,072
Accomodation 331,626 567,984 906,786 1,806,396
Total, excluding GST/HST 4,294,000 6,701,000 11,330,000 22,325,000
Table 2. Summary of CPI and CPI-EI Budgeted Expenditures, 2008/2009 to 2012/2013 by CPD (Total)
Budgeted expenditures, CPI and CPI-EI Fiscal year
2008/2009 2009/2010 2010/2011 2011/2012 2012/2013 Total
($)
Consumer Price Index 7,262,397 7,489,525 11,393,319 12,944,270 13,524,767 52,614,278
Salary 5,973,222 6,428,145 9,316,130 11,226,958 12,158,993 45,103,448
Non-salary 1,289,174 1,061,380 2,077,189 1,717,312 1,365,774 7,510,829

3. Evaluation Methodology

3.1 Evaluation Scope, Timing and Quality Control

This evaluation looked at the impacts of the CPI and the CPI-EI. The evaluation objective is to provide credible and neutral information on the ongoing relevance and performance of the CPI. The evaluation focuses on program performance from 2008/2009 to 2012/2013 for the CPI and the CPI-EI, which accounts for a total budgeted expenditure of $52,595,279.

The approach was to review and confirm the conditions before the CPI-EI was approved, in December 2010, and assess whether the key risks and weaknesses identified in relation to CPI are being addressed through the CPI-EI and whether the CPI is on track to achieve its expected outcomes. The evaluation is based on a quasi-experimental design: assessment is focused on both a retrospective and future view. The evaluation risks, availability of performance measures, evaluation funding and timing were all factors that helped calibrate the scope of this evaluation. The evaluation scope was defined accordingly to ensure an appropriate balance between level of effort and evaluation context.

The evaluation was designed and conducted in accordance with Treasury Board (TB) Policies and Guidelines as outlined in the TB's Policy on Evaluation (April 2009) and the other components of the Treasury Board Secretariat Policy Suite. The evaluation will serve Statistics Canada's accountability requirements in relation to its Treasury Board submission: it will also provide analysis and recommendations to inform future decision-making.

The evaluation was conducted from late August 2012 to March 2013. Intergage Consulting Group Inc., an Ottawa-based consulting firm, was contracted to perform the research. Intergage worked collaboratively with Evaluation and Professional Practices Division at Statistics Canada.

In addition, selected experts from the current Price Measurement Advisory Committee reviewed and provided feedback on the evaluation work plan and the final evaluation report.

Finally, validation and approval sessions for the final evaluation report took place with program representatives and Statistics Canada's Departmental Evaluation Committee.

3.2 Evaluation Questions by Issue Area

The questions for this evaluation were selected based on CPD and CPI-EI logic models. The questions and indicators by core issue are set out in the evaluation evidence matrix, and are outlined in Table 3. A more detailed table with sub-questions and indicators is in Appendix C.

Table 3. Overview of Evaluation Issues/Questions and Indicators
Issues Questions
Continued need for the program
  • Does the program continue to address a demonstrable need?
Alignement with government priorities
  • How well is CPI aligned with Statistics Canada outcomes and Government of Canada priorities?
Consistency with federal roles and responsibilities
  • Are the program activities aligned/congruent with Statistics Canada's jurisdictional, mandated and/or legislated role?
Design and implementation
  • Are the design decisions made under the CPI-EI relevant to the CPI and the context?
  • Is the CPI-EI being implemented as planned?
  • Are the right governance mechanisms/arrangements (including CPI and EI project governance) in place to support oversight and risk mitigation?
Achievement of outcomes
  • To what extent did the program achieve its expected immediate outcomes?
  • To what extent did CPI achieve its expected final outcome?
  • Are there any unexpected impacts/outcomes from the CPI-EI?
Demonstration of efficiency and economy
  • Is the program managed cost-effectively?
  • Is there a more effective approach to achieving CPI outcomes?
  • Is the CPI Program duplicating or complementing existing programs/initiatives?

3.3 Methods

3.3.1 Preliminary Consultation

Prior to undertaking the evaluation, preliminary interview consultations were done with CPI leaders. This led to the validation of the logic model and the development of an evaluation work plan, including the methodological approach.

3.3.2 Lines of Evidence

Data for this evaluation were collected from multiple sources incorporating four lines of evidence: a literature review, document and jurisdictional reviews and interviews with key informants. Logical and statistical analysis was used and findings were correlated to identify trends and patterns. The evaluation team synthesized and analyzed the preliminary findings related to each line of evidence into a summary matrix.

Literature review: The review covered 21 pieces of literature, and examined national and international academic sources, as well as studies produced by stakeholder organizations. This review explored the need for a CPI; some of the key issues and challenges for CPIs; and general trends and recommendations for addressing key CPI challenges and weaknesses.

Documentation review: This review included two categories of documents: broader government-wide documents, including appropriate acts; and Statistics Canada's program-specific documents. The review of the program-specific documents provided information on the rationale and continued need for the CPI and CPI-EI and alignment with overriding government priorities and Statistics Canada's legislative mandate; basic financial information on the CPI and the CPI-EI; as well monitoring and reporting documents in relation to the overall CPI-EI plan. The review of these documents informed evaluation issues concerning overall performance of the CPI-EI.

Jurisdictional and benchmarking review: As part of this review, a high-level comparison of the CPI approaches of six other countries—Australia, the United States, France, Netherlands, New Zealand, and the United Kingdom—to that of Canada was done pre- and post-CPI-EI. This review included comparison of different CPI baskets (e.g., number of stores, frequency of updates), sample size, distribution of sample, quality adjustments and revision approach.

Key informant interviews: 28 stakeholders from various key informant groups were interviewed. They were:

  • program representatives involved with CPI production and release, including senior leaders, chiefs and subject-matter experts from CPD, those leading system upgrades, methodology and collection changes, as well as representatives from Statistical Information Systems Division and the Survey of Household Spending — 15 individuals were interviewed in 12 meetings
  • end users of the CPI, including representatives of the Bank of Canada, Finance Canada and other interested federal departments; representatives from provincial finance ministries (eight including one internal end user); and a national association with an interest in the CPI
  • five Canadian and non-Canadian internationally recognized experts.

3.3.3 Methodological Limitations

Below are some of the key challenges and limitations in relation to the evaluation of the CPI.

Program size and complexity in relation to the evaluation budget (calibration)

The CPI-EI is a large transformational initiative that involves large budgets and diverse stakeholders. Ensuring appropriate coverage throughout this evaluation was a challenge — particularly given the budget. In particular, the number of interview consultations was relatively small given the size of the program and the complexity of stakeholders involved in the program. In some cases, it was a challenge to draw conclusions from the interviews given the low number of interviews from stakeholder groups.

Mitigation strategy: To mitigate this limitation, the evaluation team carefully selected the key informant interviewee sample to ensure it represented a balance of stakeholders who could provide informed feedback. Additional resources were freed up to support a more extensive jurisdictional review.

Focus between the program and the enhancement initiative

While the evaluation is on the CPI and the CPI-EI components, the evidence relies heavily on the CPI-EI component. This could be explained by program's high focus on the integration of the CPI-EI components since 2010/2011.

Mitigation strategy: While two logic models were developed for CPI and CPI-EI, the evaluation team decided to assess the program according to the CPI logic model. The team used the CPI-EI logic model, which is mainly output oriented, as an input for the CPI stated outcomes. This integrative approach helped assess the extent to which the program addresses needs and is helping to achieve CPI's expected outcomes.

Diversity of key informant interviews

Interviewing a variety of key informants posed a challenge: often there were only one or two interviewees from each key informant group. Each group indicated a range of opinions and needs that were sometimes quite different from others in the evaluation. Thus, reporting findings was often difficult because, in a number of areas, the feedback of key informants did not coalesce around common views. This is a fundamental challenge of the CPI: some needs of key stakeholders are mutually exclusive.

Mitigation strategy: The evaluation team corroborated key stakeholders' feedback with findings from the literature review to ensure broader support for the suggestion being made. In addition, this final evaluation report includes the key informant findings at the broader level and not at an individual level.

Potential biases of key stakeholders

Given that this is a highly specialized area, the key stakeholders consulted likely have inherent biases.

Mitigation strategy: The evaluation team used an independent quality assurance process for the project findings to ensure that no individual perspective would drive the findings of the evaluation, but that the findings were based on multiple lines of evidence. Regular internal review meetings were held to ensure that the data were captured in the evidence matrix and that there was no bias from the views of any one interviewee or member of the evaluation team.

4. Findings — Relevance

This section looks at the overall relevance of the CPI and the CPI-EI, and considers three core issues: the need for the CPI and CPI-EI, alignment with government and departmental priorities, and the congruency with role of the federal government.

4.1 Core Issue 1: Continued Need for the Program

Question 1: Is there a continued need for the program?

Does the program continue to address a demonstrable need?

Key Findings

A continued need for the CPI

The evaluation found evidence of continued need for the CPI. The index is a critical measurement of price changes used by government and other key stakeholders to inform economic and monetary policy, including development of the GDP and informing the Bank of Canada in measuring rates of inflation. Moreover, internationally the CPI is one of the most widely-used statistics for measuring price changes and economic performance.

Area of unmet needs

Although they were not consistently identified by all groups, some areas of unmet needs were identified such as sub-indexes (e.g., basic necessities index), deeper level of data (e.g., core CPI for provinces) and, post-publication revisions to the index.

4.1.1 A Continued Need for the CPI

A review of program documents and the literature, as well as consultations with key informants, showed that the CPI continues to address a demonstrable need. The CPI is a critical index that is used around the world to track and measure price changes.

The CPI is widely used as an indicator of the change in the general level of consumer prices or the rate of inflation. It is used in three ways in Canada:

  • SNA deflation—to transform nominal variables into real variables, for example, to distinguish real changes in GDP from those due to inflation
  • indexation—to compensate individuals for price changes, for example, to adjust the value of government transfers to individuals
  • inflation target—to inform monetary policy made by the Bank of Canada, the CPI feeds into the calculation of inflation and interest rate.Footnote 1

Internal key informants and documents attested to the fact that the price-level information in the CPI is vital to a wide range of personal, government, and business decision makers.

Within Statistics Canada, the CPI is important, even central to other agency activities. For instance, the CPI is a key input for macro-economic programs to monitor current economic conditions, such as a deflator for the GDP index for the System of National Accounts.

Externally, the CPI is an important index used by the Bank of Canada to help set interest rates. The Bank also uses a Core CPI published by Statistics Canada.Footnote 2 The Bank monitors core inflation to help inform inflation targeting, not as a replacement for the CPI.Footnote 3 The Bank of Canada also maintains an inflation calculator, which presents the impact of inflation on purchasing power, based on monthly CPI figures dating back to 1914.Footnote 4 As a Bank of Canada report mentions, the CPI is used extensively for these purposes because it is available on a monthly basis, it has a short publication lag, and it relies heavily on retail transaction prices rather than imputed prices.Footnote 5

The CPI is also used by Finance Canada to determine transfer payments to provinces and territories and by other government departments as an escalator for federal programs. Key informants also noted that the CPI is used by government representatives as a good reading on the pace of inflation in Canada and to inform various policy and budget decision-making advice and recommendations related to tax policy, labour market decisions as well as structural efficiency and inefficiency. The CPI is also used by governments and other sectors to adjust pensions, income taxes and contracts, such as wages, rents, and leases, as well as by universities for research.

The uses of the CPI affect all Canadians at some point in their life. For example, the CPI is used to index the Old Age Security, which impacts more than 4 million people every three months and the Canada Pension Plan, which impacts more than 5 million people every 12 months.

Internationally, the CPI is one of the most widely-used statistics for measuring price changes and economic performance. For example, the International Monetary Fund (IMF), of which Canada is a member, uses the CPI as a tool to assess countries' economic and financial development and to ensure that countries pursue policies that are conducive to orderly economic growth and reasonable price stability. According to internal key informant interviewees, CPI is also used by the OECD, in spatial indexes, and by the United Nations.

Areas of unmet needs

According to evidence found in the document review and comments from organizational end users, criticism from the general population about the CPI comes from consumers and those on fixed incomes, particularly seniors, who compare movements in the CPI to changes in their personal income to monitor and evaluate changes in their financial situation.Footnote 6 This is one of the fundamental challenges of the CPI. Criticism may result from a misunderstanding of the CPI. It is not a cost-of-living index: for example, it does not adjust to changing consumption patterns in such a manner as to leave consumer utility constant. Furthermore, given that it is a basket of average prices, it may not always be reflective of the realities of individual Canadians, depending on their demographic and geographical location. In fact, some compelling academic research concludes that the CPI is not, and can never be, a perfect index.Footnote 7,Footnote 8,Footnote 9

While those consulted understand the limitations and support the primary areas of change under the CPI-EI, there continues to be areas of unmet needs.

Organizational end users indicated an interest in developing and issuing sub-indexes to the CPI, particularly in the areas of seniors and those living close to the poverty line (e.g., a basic necessities index). In fact, the need to better reflect the actual cost of living for some age cohorts was also raised in a number of academic publications.Footnote 10 CPD has recognized that a single CPI cannot serve all uses equally well, and has been examining publishing 'families' of indices.

Furthermore, consultation with internal key informants did not yield consensus on the extent to which CPI is supporting the critical needs of key stakeholders. While some noted that the CPI is meeting stakeholders' critical needs, particularly in relation to improved basket, sample and quality assurance, others noted that stakeholders are often interested in a deeper level of data than Statistics Canada is releasing (i.e., a core CPI for provinces).

CPD has identified some of the following unmet needs in relation to the CPI:Footnote 11

  • the extent to which series are seasonally adjusted
  • the treatment of owned accommodation
  • whether a resale house price index should be included
  • whether a domestic or national perspective is adopted
  • whether there should be a superlative index
  • whether the CPI should be revised after it is published.

Experts and end user key informants also mentioned these areas of unmet needs, although they were not consistently identified by all groups. The experts were the only key informants to identify the need for a superlative index and the possibility of post-publication revisions to the index; however, the high costs associated with these changes were also noted as a challenge.

4.2 Core Issue 2: Alignment with Government Priorities

Question 2: Does the program align with Government of Canada priorities and departmental strategic outcomes?

How well is CPI aligned with Statistics Canada outcomes and Government of Canada priorities?

Key Findings

The CPI is aligned with Statistics Canada outcomes and Government of Canada priorities.

The evaluation found evidence that the production of the CPI is one of the ongoing priorities of Statistics Canada and is aligned with the federal government's priorities. The federal government requires a measure of price changes in order to inform economic and monetary policy and it is used by all levels of government.

The CPI is aligned with departmental outcomes and priorities. According to the 2012/2013 Program Alignment Activity, the program resides within the sub-activity Economy-wide Statistics of the Economic Statistics program activity. The CPI's objective is to reflect changes in consumer prices experienced by Canadians in an accurate, timely and relevant manner with the intent to support a variety of key policy programs (monetary, fiscal and social), as well as SNA requirements. The CPI objective is aligned with Statistics Canada's ongoing strategic outcome, outlined in its 2012-13 Report on Plans and Priorities:

Canadians have access to objective, high quality, non-partisan statistics, statistical products, services and analyses on Canada's economy and society which fulfill legal requirements, are relevant to policy formulation and decision makers and are responsive to emerging issues.Footnote 12

The 2012-13 Report on Plans and Priorities also mentions that "producing a comprehensive program of macroeconomic statistics to support fiscal and monetary policy is one of Statistics Canada's fundamental responsibilities." The CPI, as a main program to support fiscal and monetary policy, is considered part of Statistics Canada's priority.

The evaluation found that the CPI is aligned with Government of Canada priorities. While speeches from the throne do not specifically refer to the CPI, since 1999 they have stressed the importance of improving Canadians' quality of life and strengthening the economy. A review of federal budgets since 2006 supports the evidence of the throne speeches, demonstrating the use of the CPI to measure inflation and provide evidence to support government-wide financial decisions.

As stated in the 2012/2013 whole-of-government priorities, the federal government is committed to evidence-based decisions-making to support Canadians and other key stakeholders in various sectors of the economy:

  • A transparent, accountable and responsive federal government: "Program activities aim to safeguard public trust in government and strengthen the interaction between the government and Canadian citizens. This is done through the delivery of services that are responsive to citizens' needs, and through public accountability, compliance and recourse measures."
  • Strong economic growth: "Programs aim to create an environment conducive to economic growth and to promote the development of all sectors of the economy and in all regions of Canada."

The CPI supports federal government priorities for an accurate and timely index to inform government decisions by providing relevant pricing information, either in its original form or transformed through economic modeling to estimate future price movements. It also responds to the priority of transparency by providing objective and non-partisan statistics.

According to evidence from the key informant interviews, the jurisdictional review, and the literature review, the CPI is the most visible and credible measure to anchor inflation expectations, and it should be the basis for the measurement of core inflation. Greater accuracy in the CPI is well aligned with the priorities of Statistics Canada and the federal government overall, including the importance of having an accurate price index to inform the Bank of Canada and overall government policy.

4.3 Core Issue 3: Consistency with Federal Roles and Responsibilities

Question 3: Is the program aligned with federal roles and responsibilities?

Are the program's activities aligned/congruent with Statistics Canada's jurisdictional, mandated and/or legislated role?

Key Findings

There is a legitimate role and responsibility of the Government of Canada in producing a CPI, and for it to be produced by Statistics Canada.

There is evidence that CPI activities are congruent with Statistics Canada's legislated role and the federal government's jurisdictional role. The evaluation has found that Statistics Canada is the only organization in Canada that can produce the CPI, given its status as an independent agency, as well as its expertise and capacity.

All data collected and published by Statistics Canada is done under the authority of the Statistics Act. Even if the CPI is not stated specifically in this Act, Statistics Canada, as an agency, has the duty under sections 3a, 3b, and 3e to

  • a) collect, compile, analyze, abstract and publish statistical information relating to the commercial, industrial, financial, social, economic and general activities and condition of the people;
  • b) collaborate with departments of government in the collection, compilation and publication of statistical information, including statistics derived from the activities of those departments;
  • e) generally, promote and develop integrated social and economic statistics pertaining to the whole of Canada and to each of the provinces thereof and to coordinate plans for the integration of those statistics.Footnote 13

Other acts and regulations make reference to Statistics Canada and the Consumer Price Index. If those references do not legally require that Statistics Canada produce statistics, they assume that Statistics Canada does in fact produce such statistics as part of its regular program and require that such statistics be used for specific purposes:

  • Under section 5 of the Bretton Woods and Related Agreements Act,Footnote 14 Canada, as a member of the IMF, is required to provide statistics on price indexes and follow the Special Data Dissemination Standards (SDDS) agreement.Footnote 15 The SDDS identifies four dimensions of data dissemination. One of them, integrity of disseminated data, requires that, "official statistics must have the confidence of their users. In turn, confidence in the statistics ultimately becomes a matter of confidence in the objectivity and professionalism of the agency producing the statistics. Transparency of its practices and procedures is a key factor in creating this confidence."Footnote 16
  • The CPI is mandated under the Old Age Security Act.Footnote 17Under different sections (e.g., sections 2, 7, 22), the act requires that pensions are adjusted quarterly based on the CPI for Canada, as published by Statistics Canada under the authority of the Statistics Act.

Evidence from the jurisdictional review and key informant interviews shows that Statistics Canada is the only organization in Canada that can produce the CPI based on its status as an independent agency and its expertise and capacity. In all the jurisdictions reviewed for this evaluation, the CPI is produced monthly by the national, independent statistical agency or, in the United States, by the Bureau of Labor Statistics. All internal and end users interviewed agreed that Statistics Canada is the only organization in Canada that can produce the CPI. Statistics Canada is seen as an independent government agency with a high level of credibility, neutrality and trustworthiness. Statistics Canada also has a significant statistical and analytical infrastructure in place and, therefore, the capacity to produce the index.

5. Findings — Design and Implementation of the CPI-EI

This evaluation issue was included in the study because senior management wishes to examine critical design and implementation areas of the CPI-EI project three years after its initiation. Their reason for this examination is the critical significance of the CPI and the magnitude of resources invested in the CPI-EI.

Specifically, the evaluation study assessed the following aspects of the project implementation:

  • CPI-EI design decisions and their relevance to CPI
  • whether the implementation of the CPI-EI is advancing according to plan
  • whether the CPI-EI has appropriate governance structures.

This section presents the key findings of each of these areas of examination. Challenges applicable to the CPI-EI implementation that were identified by the evaluation are summarized at the end of the section.

5.1 CPI-EI Design Decisions

Question 4: Is the CPI Enhancement Initiative being implemented as expected?

Are the design decisions made under the CPI-EI relevant to the CPI and the context?

Key Findings

The CPI-EI was necessary to address known weaknesses of the CPI and to ensure that the index continues to be a high quality and reliable index. Evidence demonstrates that the design of the CPI-EI is sound and the activities underway are the right ones to improve the CPI.

The CPI is a highly complex economic and monetary measure. The changes to the CPI-EI — specifically more frequent basket updates, more monthly price quotes collected and quality adjustments— are appropriate measures to reduce the three critical biases observed in CPIs (e.g., substitution, new goods and outlets and quality).

The CPI-EI planned improvements are well aligned with international practices. In particular, moving to a two-year cycle for updating the basket brings Canada in closer alignment with leading jurisdictions.

Evidence from the documents reviewed, as well as all internal interviewees consulted, shows that, over the past few decades, several emerging factors necessitated renewing and improving the CPI, such as

  • More knowledgeable end users with increasing demands for a CPI that evolves, better reflects price movements and remains current with international best practices.
  • Federal downsizing in the 1990s, which led to funding and program cuts, including cuts to the CPI's sample size and increased risk of errors — for instance, an error in the Traveler Accommodation sub-index became public knowledge in 2006. This error increased public and stakeholder perception of the possibility of further errors in the calculations of the CPI.
  • Increasing criticism that Canada's CPI (as well as those of other nations) overstates inflation due to a measurement biasFootnote 18 and does not adequately reflect factors such as new emerging items, substitutions and quality measures. Concern has also been heard that Canada's CPI may not be keeping pace with those of international partners.

To reinforce the CPI and address known weaknesses, CPD undertook extensive research and consultation to find the best approach to produce a better measure of price movement: this led to the CPI-EI project. The intended goal of CPI-EI is to modernize, consolidate and streamline the CPI's processing systems to make them more efficient, coherent, consistent, secure and robust. This would prepare these systems to support planned program changes related to sample size, sample design, basket updates and overall quality, including quality adjustment (detailed in Section 2.2).

Criticism of Canada's CPI was consistent with the findings of the US Boskin Commission Report,Footnote 19 a special inquiry into the use of the CPI as a measure of the cost of living in the US. The Boskin Commission found that the overstatement of the CPI was due to biases: the important unintended consequences include over-indexing government outlays and tax brackets as well as increasing the federal deficit and debt. In Canada, the total CPI measurement bias is estimated by the Bank of Canada to be approximately 0.5 percentage points per year over the 2005-to-2011 period; similar with earlier findings estimated this bias at 0.6 percentage points.Footnote 20,Footnote 21 Slightly less than half of this bias was seen to be caused by the fixed nature of the CPI basket of goods and services (i.e. which is related to the frequency of basket updates).

A review of the literature and research provides details on the changes that were required to the CPI. First, the US Boskin Commission ReportFootnote 22 confirms that, in many countries, the approach to the CPI must improve for it to be useful as a measure of changes in the cost of living. Further, the literature contains an empirical rationale for Statistics Canada's decision to undertake the CPI-EI: in particular, it shows the importance of reducing measurement bias from traditional sources. For instance, research undertaken for the US Bureau of Labor Statistics (BLS) affirms the importance and direction of the changes proposed and under way with the CPI-EI.Footnote 23  Evidence from the literature review shows that the general policy direction of Statistics Canada's CPI-EI is in line with the Boskin report recommendations and is consistent with international trends. Updating the basket more frequently and improving the mix of products and services in the basket to better reflect what consumers are buying (all key elements of the CPI-EI) are some of the most effective approaches to addressing the upward bias in the CPI and to addressing issues of quality and speedier changes to the composition of baskets.Footnote 24,Footnote 25,Footnote 26

The jurisdictional review and consultations with internal representatives and experts found further evidence that the CPI-EI planned improvements are well aligned with international practices. In particular, moving to a two-year cycle for updating the basket aligns Canada more closely with leading jurisdictions such as France, the Netherlands, the United Kingdom, the United States and New Zealand. The first three countries update their baskets every year; the United States, every two years; and New Zealand, every three years. This further supports a conclusion that the design decisions under the CPI-EI are highly relevant.

Along with these findings, key informants agreed that the CPI-EI project seems to address the most urgent needs of improvement for the CPI and addresses the potential risks of delays and future mistakes. According to some interviewees, the CPI-EI will bring Canada back to the leading edge, internationally, in terms of these types of price indexes. Many respondents shared their expectation that the CPI-EI is only the first step and that more initiatives will need to be put in place by Statistics Canada to improve the CPI and sustain the capacity and knowledge developed during this project.

5.2 CPI-EI Implementation

Question 4: Is the CPI Enhancement Initiative being implemented as expected?

Is the CPI-EI being implemented as planned?
  • What are the key challenges related to implementation and have mitigation strategies been put into place?

Key Findings

There is evidence that the CPI-EI is being implemented as planned and is progressing well.

Evaluation evidence indicates that the CPI-EI is being implemented as planned with some areas being implemented ahead of schedule (such as increasing the timelines for basket updates and price quotas).

Evidence found in the various CPD progress reports and briefings show that the CPI-EI is being implemented as planned. The evaluation found that the progress reports are consistent with initial planning and schedule for implementation. For example, the September 2012 CPD briefing to the Chief Statistician notes that the scope of the CPI-EI remains unchanged; that deliverables are on time or advanced; and that spending is on target. Appendix D summarizes the key activities and outputs of the CPI –EI, along with the targets and details about the actual levels for the first three years of the initiative.

Some accomplishments are worth highlighting.Footnote 27

Increased sample size

  • The number of monthly price quotes reached 9,837 in Year 3, ahead of the expected 6,000.
  • In Year 3, the total price quotes in the sample were 72,000 (the baseline was 60,000 in 2009). The program is on track for its goal and will probably doubling the size of the sample to 120,000 by 2014/2015.

Improved sample designFootnote 28

  • In 2012, the program created a geographical and outlet frame for outlet selection, using information from Census, the Business Register, and the Quarterly Retail Commodity Survey.
  • The CPI sample was linked to the Business Register (the matching rate for CPI outlets was 95%).
  • The number of new outlets used in data collection for the CPI has increased from 88 in Year 1 to 584 in Year 3.

Basket updates

  • Frequency has moved from four to two years, one year ahead of schedule. In June 2011, the program successfully introduced the 2009 reference year basket update, which is the first biannual update of basket weights in the history of the program and ahead of the 2012 schedule for biannual updates. In March 2013, the program introduced the 2011 reference year basket updated.
  • Quality was improved by applying new quality adjustment methods to over 100 products ahead of the target of 75 products.
  • Progress was made to improve the timeliness of the basket updates.

The improvement of IT enablers is also progressing on track.

  • The Field data collection device was replaced in April 2011.
  • The program has streamlined field data processing and sample management system interfaces.
  • Paper forms reliance was reduced in the fall of 2012.
  • A representative product module was introduced.
  • Outlet acceptance/rejection functionality was introduced.
  • The Index Aggregation System development (Cygnus), a three-part project, completed the alpha release of its first part on July 6, 2012, and is on schedule to be completed by the target date of September 2014.

5.3 CPI-EI Governance

Question 4: Is the CPI Enhancement Initiative being implemented as expected?

Are the right governance mechanisms and arrangements (including CPI and CPI-EI project governance) in place to support oversight and risk mitigation?

Key Findings: CPI-EI Governance

The governance arrangements are viewed as appropriate and sufficient to support the CPI-EI and the ongoing management of associated risks.

CPD senior management demonstrated ability to address emerging issues with overall project management, system integration and governance by enhancing human resource capacity, implementing relevant organizational changes and improving governance structures.

Evaluation evidence revealed that structures and processes are in place to monitor implementation progress of the CPI-EI. The Price Measurement Advisory Committee (PMAC) permits Statistics Canada officials to benefit from external, independent advice regarding important statistical matters.

However, a few challenges related to governance, capacity building and continuity planning were identified, such as horizontal discussion of cross cutting issues.

5.3.1 Governance

A CPI-EI governance documentFootnote 29 details the governance structure that supports the CPI-EI (Appendix B). Documentation provided and consultations with internal key informants show that the CPI-EI governance arrangements are reasonable: they are also adequate to support the implementation of CPI-EI and to manage associated risks. Interviews with key internal informants and selected external experts confirmed that the CPI-EI governance structures are in place and working well.

In addition to its governance structures, CPD has an advisory body, the PMAC. At its annual meetings, PMAC plays the critical role of bringing international expertise and best practices to inform Statistics Canada's approach to the CPI; acting as a sounding board for new ideas; and performing a challenge function. According to some key informants, however, there is room for the PMAC to improve, "by focusing on the setting of the agenda, asking the right questions, and narrowing down priorities and trade-offs."

5.3.2 Project Management

"To instill more project management rigor is a project in itself in CPI. It is becoming a dynamic program and will change over time. We are building capacity for this in the CPI. This is taking time. From my perspective, it is working and we are on target. Theory is now backed up by practice."

Internal informant

Being a highly specialized division, CPD had significant statistical expertise but needed "more planning expertise to deliver on time and budget." At the beginning of the CPI-EI, the practice in the division was more reactive than strategic (e.g., "to do more acting than planning.") CPD invested in project management training, in particular the Project Management Body of Knowledge (PMBOK) principles for CPD managers. This investment was viewed as taking time to develop capacity, but most internal informants believe that it is working and that the program management practices have improved: the PMBOK principles provide rigor and objective measures for monitoring project status, focusing on effort expended (time, budget, and resources).Footnote 30 This, internal informants attest, has helped build a strong project management capacity and culture. Examples of relevant management practices are presented in Appendix E.

Program documents show that, as of 2012, CPD has adopted the Departmental Project Management Framework (DPMF) and has developed several tools to facilitate the planning process and to provide inputs into the DPMF templates.

5.4 Implementation Challenges

The evaluation study identified a few governance and implementation challenges.

5.4.1 Governance Challenges

Interviews with internal key informants provided evidence of some challenges in setting up the CPI-EI governance structure and ongoing project management. The primary reason raised was that, operationally, the CPI-EI was set up and funded as a single project; however, the project actually consists of six inter-related yet independent sub-project groups.Footnote 31 Further challenges were noted:

  • Governance arrangements do not seem to be well understood by all CPD staff, particularly those at lower levels. This is also related to the need to further clarify areas of responsibilities and hand-offs in areas such as sample design, QA and basket updates.
  • A few internal informants noted that there were no effective mechanisms for bringing cross-cutting issues to the table for discussion at lower levels in the organization. They made reference to the previously existing Issues Committee, composed of economists, where areas of interest to the CPI such as research, and findings and recommendations used to be presented to management. These informants suggested that there continues to be a need within CPD for a horizontal mechanism at the chief, director, senior researcher and economist levels to discuss cross-cutting issues. This may have been addressed with the recently established practice of weekly integration meetings.

5.4.2 Capacity-building and Continuity-planning Challenges

Given the level of complexity of the CPI and the CPI-EI, internal key informants noted that CPD requires an expanded body of professional expertise in the theory and practice of price index numbers. This skill set is relatively scarce; few universities have graduate programs to address it. At the launch of the initiative, CPD needed to recruit additional personnel from within and outside Statistics Canada, and then develop these resources to meet the project needs. The hiring process and subsequent learning and development initiatives took investment of time and effort during the first two years: this investment is expected to yield substantial returns in subsequent year of the project. Program progress reports indicate that these activities have succeeded in building capacity and have not slowed project implementation.

However, several internal informants indicated concern that the CPD has lost significant subject-matter expertise, primarily at the chief level and above, as a result of retirement and organizational turnover. There is concern that an erosion of subject-matter expertise could put CPI-EI implementation and CPI modernization at risk, especially with respect to succession and continuity planning.

6. Findings — Performance

6.1 Core Issue 4: Achievement of Expected Outcomes

This section details findings related to the four streams of expectedresults that the CPI is expected to achieve: data production, information dissemination, research and development and information technology.

Immediate and intermediate outcomes were assessed separately. However, given the close relationships between immediate and intermediate outcomes in each area, it was determined that the best approach would be to report them together for a more integrated view and better understanding of each stream of outcomes.

In the case of data production, only the findings for the immediate outcome are reported in the achievement of outcome section. This decision was taken to avoid duplication: findings for the intermediate outcome, "provide information that supports the needs of key programs and stakeholders" were very similar to the findings for Core Issue 1, "continued need for program."

Finally, findings for the final outcome "Canadians have an accurate, timely, relevant set of consumer price indexes meant to support a variety of key policy programs" will be presented in Section 7.

Question 5: Is the program achieving the expected outcomes outlined in the logic model?

Data production
  • Produce trusted and relevant data in a timely manner (immediate)

Key Findings

Canada's CPI meets international standards and is an index that is trusted, credible, relevant and timely.

Evidence indicated that Canada's CPI is trusted, timely (i.e., CPD has never missed a release) and is meeting international quality standards. Evidence also revealed that the changes ushered in under the CPI-EI are consistent with the approach found in other countries, and that the CPI-EI is needed to maintain continued confidence in the CPI.

Recent changes introduced with the CPI-EI enable production of more trusted and relevant data

Early results of the CPI-EI show that the level of confidence in CPI data and its representation of consumer spending is being reinforced through a better quality assurance process and a more frequent and timely update of its basket weights. CPI-EI was needed to address some weaknesses of the CPI and to ensure continued confidence of users in the CPI.

Data production encompasses processing price quotes, calculating price indexes, analyzing data as part of the quality assurance processes, and liaising and co-ordinating with partners who collect statistical information. CPD produces temporal and spatial consumer price indexes. Production of each of those product lines involves collecting, processing and assessing the information. The documentation review and key informant consultations indicated that Canada's CPI data is trusted and timely.

Almost all key end users and experts indicated that the CPI is a trusted and credible price index. Some respondents affirmed that this trust comes from Statistics Canada's strong reputation in the area of price indexes, the Agency's access to good information on best practices (through advisory committees, and participation at international meetings) and its implementation of leading edge practices through the CPI-EI. A few end users expressed reservations due to known biases evident in the CPI and known past errors in the calculations of the CPI. Key informants reported that, while the trust in the index could be higher they highly trust the index because they are aware of its limitations. Consultations with key informants further revealed that CPD has never missed a release, and that the CPI cannot be produced more frequently than monthly.

There is a grey area regarding the level of trust. So we put our answer at "between medium and high" level of confidence. We know the weaknesses, and we use the CPI with knowledge of these weaknesses.

End user

This finding is reinforced by international studies. According to the literature reviewed, the CPI is considered a trusted and timely index that met international quality standards for statistical agencies and for national CPIs. As far back as 2003, all IMF reports concluded that CPI produces data that are generally trusted based on proven quality assurance practices. For instance, the 2003 IMFFootnote 32 conclusions were restated in the 2011 IMF Annual Observance Report, which concluded that the Canadian CPI meets the entire SDDS requirement under the Data dimensionFootnote 33 (Table 4), the Access dimension,Footnote 34 the Integrity dimensionsFootnote 35 and the Data Quality dimension.Footnote 36 In fact, IMF reports since 2008 conclude that, "The quality, coverage, periodicity, and timeliness of Canada's economic data are considered to be excellent both in the context of the Article IV consultation and for purposes of ongoing surveillance." (IMF, 2008, 2009, 2010, 2011).

Table 4. CPI Timeliness against SDDS Requirement, 2008 to 2011
Price Index: Consumer prices
(real sector)
National Periodicity
(Canada)
Prescribed Release Lag (SDDS) Reference Period
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
2008 to 2011 Monthly 1 month X X X X X X X X X X X X

All key informants agreed that the CPI-EI was needed to address weaknesses in the CPI and to ensure continued confidence in the CPI. According to experts consulted, the CPI-EI is expected to further improve an already good price index. Most respondents concluded that the CPI-EI is still in early stages of implementation: while these changes make sense in theory and should address weaknesses identified, more time is needed to see results of the change and IT upgrades. Nonetheless, the early findings suggest the CPI-EI is leading to the production of more trusted and relevant data and a more trusted CPI.

Question 5: Is the program achieving the expected outcomes outlined in the logic model?

Dissemination and Communication
  • Put information products into public domains (immediate)
  • Provide high quality service to users (intermediate)

Key Findings

Information about the CPI is made available in various public domains

Evidence indicated that Statistics Canada's CPI data, research, publications and methodology are made public in an open manner through Statistics Canada website even if, at times, they are difficult to find. Social media and a free data strategy recently implemented made information more accessible.

CPD appears to have well established relations with their primary clients, but have invested less time engaging other stakeholders in the CPI and CPI-EI. Informants desire better communications and dissemination of more detailed information and data and more frequent updates.

Improvements can be made in communicating information about the CPI, particularly about the CPI-EI, by providing more information and detail to the general public and by providing more frequent updates on the methods described in the CPI reference paper.

Users face challenges in finding someone in CPD to discuss complex issues and questions related to the CPI.

Statistics Canada staff are competent and have the knowledge needed to respond to specific requests. However, it can be difficult to identify the most appropriate expert at CPD who can respond to a given request or discuss CPI questions and issues.

6.2 CPI Information Available in Public Domains

6.2.1 Web-based Accessibility

The documentation reviewed showed that Statistics Canada's CPI products, research and publications are made public in an open and regular manner on its website. Table 5 shows that users with Internet access could choose from a number of products since fiscal year 1997/1998.Footnote 37 However, between fiscal years 2008/2009 and 2012/2013, only two new publications and four information documents for analysts and researchers were publicly released. This confirms some key informants' perceptions that Statistics Canada was not releasing enough updates and recent publications on the CPI.

Table 5. Related Consumer Price Index Products on Statistics Canada's Website Footnote 38
Product Number
of CPI products published and/or updated,
1997/1998 to 2012/2013
Number
of CPI products published and/or updated,
2008/2009 to 2012/2013
Articles and reports
Publications1 31 2
Information for analysts and researchers 19 4
The Daily2 174 62
Data tables
Summary tables 35 35
CANSIM 28 13
Definitions and Documentation
Definitions, data sources and methods 40 21
Questionnaires 4 4
Surveys 2 2

The main tool to disseminate and inform Canadians on the CPI is The Daily. This document is a record of the first official release of new datasets and products, and is also used to inform the public of the significant and noteworthy findings contained in the released CPI datasets and analytical products. To date, The Daily is the most efficient way to meet IMF standards for data releases. Published in conjunction with The Daily, the Consumer Price Index module is amonthly publication, geared toward educated laypeople and expert users on CPI data. It offers a more detailed monthly analysis for each CPI component, with accompanying charts, tables and briefing notes.

According to internal data, during fiscal year 2011/2012 the CPI module in the Daily was the most viewed part of the Statistics Canada site, with a total of 515,958 viewers.Footnote 39 End users consulted indicated that more information could be made available through The Daily such as monthly variation and rolling averages across different time periods.

Users can also access detailed CPI data from Summary tables, CANSIM and ESTAT. CPI data are loaded on CANSIM, a database that is updated daily. Until 2012, E-STAT was largely used to access CPI data. However, since February 1, 2012, all Statistics Canada's self-serve standard products, including CANSIM data, became available free of charge in its website, rendering E-STAT no longer relevant. E-STAT is no longer updated and will be discontinued July 1, 2013.

The availability of free data and the dissemination strategy of Statistics Canada's communications and dissemination group increased the access and availability to statistical data. Table 6 shows that publication views grew 215 percent and the viewed Summary tables grew 36 percent over the period 2006/2007 to 2011/2012. CANSIM information has accounted for more traffic since it became freely available in February 2012. The CPI data are the most popular table downloaded by CANSIM distributors for their clients.

Table 6. Views of CPI-related Pages on Statistics Canada's Website
CPI product Number of page views
2006/2007 2007/2008 2008/2009 2009/2010 2010/2011 2011/2012 2012/20131
Publications 115,916 221,625 347,642 338,791 405,413 365,614  n/a
Summary tables 215,205 210,275 294,515 318,688 337,723 294,264  419,420
CANSIM Not applicable2 12,9473 77,634

A few external end users raised issues about the accessibility of information because of the user interface of Statistics Canada's website. To find the right information, the user has to know their way around the Statistics Canada website in relation to the CPI. Some of the external end users consulted found searching for CPI-related information to be time-consuming. But the general view of all the experts and most end users consulted was that basic CPI information and products are well disseminated and communicated by Statistics Canada through its website.

Social media are also used to disseminate information on CPI. Registered followers can receive updates on CPI from Twitter (since January 2010), RSS feeds and mobile web (since March 2011), as well as Facebook (since February 2012). One user indicated that the RSS feeds increase the timeliness of the CPI data release.

The Statistics Canada CPI publications are frequently reported in the national media. According to an internal reporting document, the print, television, Internet and radio media use CPI data extensively — so extensively that it is difficult to fully assess media usage. In 2008/2009 fiscal year, there were 340 media mentions of the CPI in the 45 main news sources monitored in Newsbriefs. For fiscal year 2011/2012, the number of media mentions rose to 485 and as of March 26, 2013, the number of mention in the media for fiscal year 2012/2013 was 332 for fiscal year 2012/2013.Footnote 40

6.3 Stakeholder engagement

Consultations with senior internal key informants indicated that CPD has well defined and positive relations with their primary clients — the Bank of Canada, Finance Canada and the System of National Accounts (SNA). Interview consultations revealed that CPD meets regularly with representatives of the National Accounts. CPD representatives also meet with the Bank of Canada through high-level bilateral meetings at least twice a year, and with Finance Canada as needed. Beyond these primary clients, CPD's relations with other stakeholder groups such as other federal departments, provincial and territorial governments and interested national associations (e.g., pensioners) appear to be less frequent. Although Statistics Canada meets annually with provincial and territorial representatives, it does so on a broad consultative agenda: the CPI is one of the topics usually covered.

End users generally recognize that the CPD is attempting to improve how it presents CPI issues and changes to key stakeholders; however, most end users indicated that more could be done to increase knowledge exchange between CPI and key stakeholders.

Internal key informants recognize a gap in CPD's approach to stakeholder engagement and consultations. However some internal key informants caution that the division needs to balance expectations of Statistics Canada's neutrality and independence and the expectation to engage key stakeholders more broadly in determining the composition of the CPI.

6.3.1 More detailed information expected

For particular needs, end users consulted expressed appreciation that Statistics Canada now offers greater access to the CPI microdata:Footnote 41 this access enables them to run a deeper level of analysis. Internal key informants noted that if CPI information related to the above-mentioned access is released to the public, the underlying confidential data are not made public.

Key informants consulted indicated that the CPD could do a better job of communicating CPI information. External informants indicated a need for Statistics Canada to explain the CPI in simpler language that the average person would understand and, ultimately, for end users to explain the CPI to various stakeholder groups. The educational approach of the Bank of Canada website was raised by an expert as a good example to follow. Specific areas identified include, what does the CPI measure; how is it developed; what does it mean to Canadians; and how does Canada's CPI compare to CPIs of other jurisdictions. Better, simpler information on the CPI would generate efficiencies by reducing or eliminating the need for CPD and other government end users to spend time explaining the CPI to those who enquire.

More frequent updates on the CPI and the CPI-EI are also needed. According to key informants, little is known publicly about the CPI-EI to the point that other than CPD's primary clients — other end users knew little about the CPI-EI or its progress. Most users interviewed suggested that updates on CPI-EI should be shared via the website. The CPI reference paper and updates on the methodology used is also an issue for end users. A few key informants noted that the reference paper is dated and would require frequent updates to ensure it reflects changes in calculations and current practices at Statistics Canada.

Informants suggested that more open communication between CPD and the public may help reduce mixed messages and misinterpretation on the validity of the methodological approach, and may help key stakeholders defend Statistics Canada data and methodology.

6.3.2 Quality of Service Support

"Good information is provided but, often, it is difficult to get in contact with analysts if you have questions about data. It takes time to find the right person at StatCan who can answer the question. There can be delays, and we face short timeframes when we have to brief our minister on the data."

End user

External end users reported that Statistics Canada staff is competent and have the knowledge needed to respond to specific requests. However, it can be difficult to identify the most appropriate expert at CPD who can respond to a given request or discuss CPI questions and issues. While calls to request help on the CPI are responded to, end users indicated that it can take up to two days for Statistics Canada to follow up. These timelines are not helpful to federal representatives or officials in provincial/territorial finance departments trying to answer an urgent question about a CPI release (e.g. minister's briefing notes).

Experts and end users indicated that it would be advantageous for Statistics Canada to publish a list of officials to contact for more information about specific technical topics such as basket composition, how the CPI is calculated, or sub-index composition and weighting. They suggested that such a list of officials who are expert in specific areas would facilitate getting CPI questions answered more promptly.

Question 5: Is the program achieving the expected outcomes outlined in the logic model?

Research and Development
  • Ensure that released data stand the test of scrutiny (CPI - Immediate)

Key Findings

There is evidence that the CPI data and index withstands public and expert scrutiny. However, there is a desire for the CPI to evolve further.

Evidence from the document review shows that Canada's CPI is consistent with internationally accepted standards, guidelines and good practices, which suggest that it can withstand independent scrutiny.

In addition, the open approach of Statistics Canada to make information available allows the public and independent parties to review and challenge the CPI, which increases its credibility. PMAC helps maintain the credibility and reputation of CPI data by providing valuable advice and a challenge function.

Suggestions were offered to keep evolving and improving the CPI to ensure that CPD keeps abreast of emerging theories and practices in some areas such as continuous evaluation of price changes and owner-occupied housing.

According to a 2003 IMF report,Footnote 42 Statistics Canada's CPI has generally followed internationally accepted guidelines and good practices, and has met the international standardsFootnote 43 (see Table 7) for several elements: prerequisites to quality, integrity, methodological soundness, accuracy and reliability, serviceability, as well as accessibility. Of these, the elements that directly pertain to research and development are methodological soundness and accuracy and reliability.Footnote 44 , Footnote 45

Table 7. Canada: Data Quality Assessment Framework – Summary of Results for Price Statistics Footnote 46
Element
Methodological soundness
Concepts and definitions Scope Classification/sectorization Basis for recording
X X X X
Accuracy and reliability
Source data Statistical techniques Assessment and validation of source data Assessment and validation of intermediate data and statistical outputs Revision studies
X X X X Not available

A wealth of detailed information regarding the CPI is available in the public domain. The CPI reference paper, various research documents on the CPI, and the CPI sub-indexes are public. CPI data are public, as well as the CPI basket, basket updates, and the weighting of the elements in the basket. This open approach of Statistics Canada enables public and independent parties to review and challenge the CPI, if need be: thus, it adds to CPI credibility.

According to internal key informants, research and publications on various topics related to the production of the CPI are published periodically. In addition, Statistics Canada officials participate regularly in international expert meetings with their counterparts in other countries. This exchange of information among international experts is seen by the some of the key informants as an opportunity to advance thinking and enable different jurisdictions to exchange views on statistical matters: this, in turn, makes it possible for CPD to benefit from practices in other jurisdictions. Along those lines, key informants acknowledge PMAC's role in providing independent expert advice and a challenge function. This demonstrates Statistics Canada's openness to outside scrutiny and dedication to meeting high quality standards, thus, contributing to the reputation of Canada's CPI.

Overall, key informants agree that the CPI can withstand the test of expert and public scrutiny. Nevertheless, they feel that there is always room for improvement, and recommend continual research and development effort to ensure that CPD keeps abreast of emerging theories and practices. Experts and end users identified areas that could benefit from further research such as continuous evaluation of price changes and their impact. Examples also include owner-occupied housing, price movements of rents, seasonal sales and automobiles. However, there was no consensus among external interviewees on which areas are of greatest priority.

Question 5: Is the program achieving the expected outcomes outlined in the logic model?

Research and Development
  • Enhance the design of the program — concepts, methods and data sources (intermediate)
    • Including CPI-EI expected outcomes

Key Findings

There is evidence that the CPI-EI is enhancing the quality of the CPI and that four areas of renewal address the key sources of bias and primary weaknesses in the CPI.

The program is well positioned to achieve its intermediate outcome, and it is progressing toward this.

The CPI-EI is achieving its outcomes: a broader range of price quotes; a more representative sample of products and stores for sampling, including more timely introduction of new products and stores to the sample, and more timely reflection of changing consumer spending patterns, as well as advanced techniques to implement quality adjustment.

Evaluation findings previously presented indirectly support the observation that CPD's research and development in recent years, particularly in relation to the CPI-EI, has led to improvements in the CPI.

More specifically,

  • Issue 4, Design and Implementation of the CPI-EI: evidence was presented that the design of the CPI-EI is sound, the initiatives underway are the right ones to improve the CPI, and the CPI-EI is being implemented as planned and is progressing well.
  • Evidence that recent changes introduced with the CPI-EI are well aligned to produce more trusted and relevant data were found regarding data production (Section 6.1)

Recent program reports confirm that, under the CPI-EI, the program focuses its research and development capacity in assuring that the CPI is accurate (e.g., represents what it seeks to measure), coherent (its component parts are integrated harmoniously), and renewable (e.g., it can perpetuate and reinvent itself over time).

For example, to increase the representativeness of the sample, the CPI-EI improved the program's 'bricks and mortar'Footnote 47 frame, to ensure the sample is "not probabilistic but rather a result of a judgmental sampling, applied in a much more sophisticated way than before." The improvements support the use of alternative inputs (such as Census, Business Register, Quarterly Retail Commodity Survey, other surveys, and third-party outlet or product databases); the use of different collection modes (including scanner data, web scraping, and internet) and the introduction of some pure Internet elements.

The coherence of the CPI has been improved as a result of reviews of key methodologies; alignments applied to CPI classifications and the aggregates' formulas; and coordinated or simultaneous updates of upper- and lower-level indexes. In the same vein, the program documents testify about stronger links and better communication with the analysis unit and the field collection staff, and about increased capacity and know-how on processing larger volumes of price quotes.

Internal interviewees specifically mentioned the research conducted to assess the impact of changing the sample size in order to broaden the sample to account for rural areas. Those consulted also describe the benefits of Cygnus, the new powerful price index estimation system, for research. The advanced IT tools of Cygnus will allow internal users and researchers to run simulations, which will help improve understanding, inform future improvements and quality adjustments. Internal key informants believe that Cygnus will open up research and analysis and will increase the accuracy of the CPI.

All categories of key informants agree that CPI-EI is on the right track to achieve its outcomes, including: broader range of prices selected, a more representative sample, more timely reflection of changing consumer spending patterns and advanced techniques to implement quality adjustment.

Expert informants acknowledge the contribution of Statistics Canada's CPI research and publications related to CPI, and suggest some areas of additional research that could generate future benefits:

  • Mini-reviews of the basket every year would allow CPD to tweak it, and are needed because the rapid change in consumer experience could not be accurately predicted;
  • Use of scanner data and web-scraping data. In this regard, experts clarify that the use of scanner data for improving the sampling is different from the use of scanner data as a data source to compile CPI, thus, different research goals and challenges are associated with each;
  • A comprehensive owner-occupied housing index;
  • Alternative data sources. In this regard, the experts reported that other countries (not only Canada) struggle with many challenges linked to the use of alternative data sources (such as issues, expenses, and problems of cleaning the data), and that so far no simple and affordable solution exists.

Question 5: Is the program achieving the expected outcomes outlined in the logic model?

Information technology
  • Ensure that released data rest on a robust IT infrastructure (immediate)
  • Enhance the IT infrastructure of the program (intermediate)

Key Findings

There is evidence that the CPI-EI has led to the enhancement of the IT infrastructure with the introduction of improvements, greater efficiencies and modernization of the CPI computer systems and tools used for collecting and analyzing the data and calculating the index.

Evidence indicated that the CPI's IT and technical systems were insufficient to meet the evolving needs of the CPI. A key focus of the CPI-EI is to improve the CPI's IT infrastructure.

Currently, it is meeting CPI requirements and is improving CPI data collection methods.

At the time of the evaluation, major progress had been made towards implementing the enhanced modules and systems, which suggests that the program is advancing in the right direction and is expected to achieve its predicted outcomes when the changes are completed.

There is some concern about the long-term sustainability of CPI's IT infrastructure and a belief that the coordination and integration of IT systems are important.

Background

The CPI is a large statistical program dependent on large, complex IT systems to execute its work. These systems cover:

  • collection: the transfer of sample specifications to data collectors in the field and the receipt of price and characteristics data and information (using in-house and corporate collection service resources);
  • processing: the statistical processing, editing and cleaning of microdata (using the Phoenix system);
  • index aggregation: the creation of indices from cleaned data and consumer expenditure weights (using the Matrix Processing System);
  • analysis and dissemination: the production of reports, custom and special tabulations and datasets.
Findings

Internal program documents provide evidence that CPI's IT and technical systems were insufficient to meet the evolving needs of the CPI. Data processing systems in CPD had not been renewed frequently enough to keep up with the evolving demands for Canada's CPI. Consultations with internal key informants provided clear evidence that significant thought was given to updating CPI's IT infrastructure to enable faster and more effective data collection, processing and dissemination.

The largest of the IT initiatives within CPI-EI is to develop a modernized price index estimation system on a stand-alone application – Cygnus, which will replace the Matrix Processing System (MPS).Footnote 48 Enhancing the collection IT systems is also critical to supporting the sample modifications and increases that are central to the CPI-EI. The other systems supporting processing, analysis and dissemination are being upgraded as needed to support CPI-EI. Although these systems are not a focus of the CPI-EI, the interconnectedness between the CPI systems necessitates that these also be considered together with other system changes.

The 2012 Progress Report briefing to the Chief Statistician provides evidence that significant progress has been made in developing and implementing the new IT enablers:

  • in collection, CPD has replaced field collection devices, streamlined field data processing and sample management systems interfaces and eliminated paper forms;
  • in data processing, CPD introduced representative product module and outlet acceptance/rejection functionality;
  • for Cygnus, the execution phase is in progress and on time; CPD released the first working functionalities in July 2012.

"Changes to IT are still very much ongoing and so difficult to assess what impact they will ultimately have on CPI. From a sampling perspective, the minimum is in place to increase sampling but changes to IT have only had a minimal impact so far."

Internal informant

As evident at the time of this report, several new or improved components are at various stages (gates) of completion. Some are fully implemented and integrated, thus it is difficult to assess progress toward achieving the CPI outcome.

Nevertheless, evidence from interviews with internal key informants suggests causal links between features of the new enabling systems and significant contributions to the functionality and other specific aspects of the production and interpretation of the CPI. For example:

  • Improvements in collection (e.g., the replaced field collection device, and elimination of paper forms) enable a more representative sample. These improvements also supported the development of the geographical and outlet frameworks.
  • Integrating the multiple and fragmented Excel and SAS software modules into the new system improves data processing functionality and the handling of some applications. The redesign of Phoenix, which is to start in 2015, is expected to improve further the consistency and integrity of the data processing.
  • The complete implementation of Cygnus enables better data navigation and analysis, as well as improved reporting capabilities: it is a huge improvement compared to the MPS.
  • The Cygnus interface, which is more user-friendly than that of MPS, reduces human error, especially on the editing side.
  • Cygnus's ability to support simulations is seen as extremely valuable, internal key informants attested. Specifically, visualization and usability tools would enable more powerful analysis. This, in turn, would build expertise and capacity in this area.

"We now know what grocery store we are collecting information from and ensure that it is in the top 20 stores."

Internal informant

Internal key informants generally agreed that the enhancements to the IT enablers in the CPI-EI support, or will support when they are fully implemented, production of a better CPI. "Better" includes more frequent basket updates, better and more efficient quality adjustments, better analysis and interpretation, better aggregation and dissemination. In the same vein, internal key informants stressed better coordination and integration of the systems across data collection, processing aggregation and dissemination.

Consultations with internal key informants and a review of planning documents indicated that CPD, as part of its evergreen strategy for the CPI, plan to renew its key systems (i.e., Cygnus and Phoenix) over a rolling five-year cycle. CPD is now focused on fully developing Cygnus; in 2015, it will shift resources to renew Phoenix. Some internal key informants expressed concern about the sustainability of CPI's IT systems and processes to support CPI, and questioned CPD's ability to systematically renew it. There is a worry that, during times of future resource constraint, the program may once again be at risk of resource reductions and operational cuts.

Question 5: Is the program achieving the expected outcomes outlined in the logic model?

  • Unexpected impacts/outcomes from the CPI-EI

Key Findings: Unexpected Outcomes

Very few unexpected impacts or outcomes were identified.

A key unexpected outcome from the CPI-EI was the increase in internal project management expertise and capacity, which is viewed as a lasting positive impact of the initiative.

A review of the documents related to the CPI-EI and key informant interviews indicated very little evidence of unexpected outcomes from the CPI-EI.

The major unexpected outcomes arise from implementing the CPI-EI outcomes. Key internal informants noted the increase in internal project management expertise and capacity, and that this will be a lasting benefit of the initiative.

6.4 Core Issue 5: Demonstration of Efficiency and Economy

In assessing the efficiency and economy of the CPI program, the evaluation team found some challenges: it required a more focused quantitative analysis, including a comparison with other similar statistics programs. Notwithstanding the limitation, resource utilization according expenditures could be assessed and alternatives to improve efficiency were identified.

Question 6: Does the program demonstrate efficient and economical decision-making and implementation?

Is the program managed in a cost-effective manner?
  • Are the resources dedicated to this program being used effectively and efficiently to maximize the achievement of outputs and outcomes?

Key Findings

The expenditures on the CPI and the CPI-EI appear to be fully in accordance with the original budget.

A review of program financial data shows that, from a financial perspective, the project is being implemented according to plan.

There is some evidence that the CPI program has efficient administrative and operational procedures. There is some opportunity to improve efficiency with internal stakeholders and by implementing a performance measurement system.

Based on the limited efficiency and economy assessment, the program seems to use an efficiency-based approach in its practices. Some improvements were suggested to better harmonize the processes between CPI and SNA/SHS.

The expenditure data for the CPI and CPI-EI revealed that spending is within budget. Overall program spending per fiscal year has remained close to budgeted expenditure, except in fiscal year 2008/2009. As mentioned by the program in their quadrennial program review, the statements reported a deficit of 10% due to overspending in salary to boost internal capacity for implementing the redesign of the processing infrastructure (Table 8).

With new funding introduced in fiscal year 2010/2011 for CPI-EI, the program was able to remain in accordance with authorities and the planned utilization rate (a range of 5%). According to internal documents, the CPI-EI faced some challenges spending its salary budget during the first two years. The challenges were the need to develop and operationalize price index expertise, and the time involved in staffing the many positions required to support the CPI-EI in a time of government downsizing.

Internal documentation highlights:

  • 33% of CPI-EI spending is IT-related in 2011/2012
  • 52% of the CPI-EI spending is directly program related to three CPI-EI drivers: basket updates, sample design and quality adjustment in 2011/2012.
Table 8. Summary of Planned versus Actual CPI Expenditures
Fiscal
year
2008/2009 2009/2010 2010/2011 2011/2012 2012/20131 Subtotal Total
2008/2009 to
2011/2012
2008/2009 to
2012/2013
$ % $ % $ % $ % $ % $ % $ %
Planned
Salary 5,973,222 82 6,428,145 86 9,316,130 82 11,226,958 87 12,139,995 90 32,944,455 84 45,084,450 86
Non-salary 1,289,174 18 1,061,380 14 2,077,189 18 1,717,312 13 1,365,774 10 6,145,055 16 7,510,829 14
Total 7,262,397 100 7,489,525 100 11,393,319 100 12,944,270 100 13,505,769 100 39,098,511 100 52,595,279 100
Actuals
Salary 6,898,279 86 6,337,765 85 8,866,273 82 11,541,278 85 8,871,188 94 33,643,595 84 42,514,783 86
Non-salary 1,119,474 14 1,074,854 15 1,947,532 18 2,054,226 15 557,088 6 6,196,086 16 6,753,174 14
Total 8,017,753 100 7,412,619 100 10,813,805 100 13,595,504 100 9,428,277 100 39,839,681 100 49,267,958 100
Variation2 (%)
Salary   -15   1   5   -3       -2    
Non-salary   13   -1   6   -20       -1    
Total   -10   1   5   -5       -2    

Review of program documents and interviews with internal key informants revealed that, prior to the implementation of CPI-EI, CPD examined various alternatives to improve weaknesses found in the CPI, including approaches followed in other countries. CPD established a working group to recommend ways to improve the CPI. The working group came up with 70 business requirements for Cygnus. This drove the option analysis and decision making to ensure that all recommendations met the working requirements and introduced the greatest efficiencies.

Internal documents show that CPD was able to meet planned deliverables for the CPI-EI: in some cases CPD exceeded targets (e.g., basket updates). Efficiency seems to be a priority for CPI management. As mentioned in an update report to the CPI-EI Steering Committee, for quality adjustment group "A cost-conscious philosophy prevails: methods are only as sophisticated and expensive as they need to be," reported an update to the CPI-EI Steering Committee.

To assess the efficiency of program administration, overhead cost is usually a good indicator. According to internal documents for 2011/2012, administration overhead was in the commonly observed range of 16% of total project spending, which includes project administration and governance, 7.2%; training, 2.3%; and production support, 6.5%.

While no inefficiencies in the operation of the CPI were found, two areas requiring further attention were identified, both concerning internal collaboration:

  • The SHS update cycle is different from CPI's update cycle. This past year, the SHS program had challenges providing data by CPD deadlines. Better integration of these cycles could improve efficiency.
  • Better aligning methodology between CPI and the SNA could yield benefits. Of interest is the integration of the product classification standard used between CPD and the SNA. CPD uses its own expenditure classification; the SNA uses the international COICOP standard. If both used the same classification, this would make data integration between the two groups easier and more efficient. A review of internal documents indicated that the concepts and methods used in CPI and SNA should ideally be as similar as possible so that early price signals provided by the CPI can be better integrated into GDP figures.Footnote 49

Performance measurement

The program has in place an accountability system through its governance structure (i.e. committee reporting). However, information is mainly technical and little information is available about the program's impact and effectiveness. Better outcome need to be defined and a performance measurement system should be set up to monitor efficiency and assess progress towards expected outcomes.

Question 6: Does the program demonstrate efficient and economical decision-making and implementation?

Is there a more effective approach to achieving CPI outcomes?

Key Findings

There is limited evidence of more effective approaches to addressing the biases in the CPI other than what has been already identified.

Alternative approaches and considerations were raised and supported by the jurisdictional review, including moving the CPI to a one-year update cycle; making use of alternative data sources and data collection methods; examining the benefits of introducing a superlative index and allowing for revisions; and using seasonal adjustments.

The jurisdictional review and consultations with internal key informants did yield evidence that the cost of data collection is a factor affecting the frequency of basket updates, sample size and restricting of samples to urban areas. Some alternatives and considerations were suggested to improve the efficiency and the quality of the CPI index.

  1. Geographic distribution of the sample — Australia, France, New Zealand and the United States draw their basket prices primarily from urban areas. Every country has unique facets of its population distribution and geography. Statistics Canada's approach to the CPI is focused on ensuring the sample adequately captures Canada's geographic diversity: this includes balancing prices taken from urban and rural areas. This is a more costly approach, but internal key informants reported that this a cost Canada is willing to incur to ensure the CPI adequately reflects prices in all parts of Canada, including rural and remote/northern areas of Canada.
  2. Moving the CPI to an annual update cycle — Experts and end users indicated satisfaction that the CPI is now being updated on a two-year cycle. However, most felt that, budgets permitting, Canada's CPI should eventually move to a one-year basket update cycle to further reduce sample bias. Most of the jurisdictions reviewed update their CPI baskets annually. Four internal informants reported that CPD envisions a one-year basket-update cycle in the future. However, when the CPI-EI was designed, a two-year update was deemed a more appropriate target.
  3. Using alternative data sources — Key informants wished to see the CPI make greater use of alternative sources of data, and alternative data-collection methods, as a way to realize program efficiencies and leverage limited program resources. For instance, Statistics Canada already buys and uses alternative data sources and is considering others, such the Internet and large insurance databases for some prices. However, internal and end user informants suggested going further with web scrapings or extractions from the Google Price Index, EBay or Yahoo, where real time price and quantity data for transactions are published. External experts indicated that using alternative data sources comes with technical difficulties and risks, and recommended that Statistics Canada carefully consider and manage the risks associated with using alternative data sources.
  4. Alternative data-collection methods Expert informants and the jurisdictional review revealed that the United Kingdom and Switzerland have outsourced the entire handheld data collection component of the CPI. There is some support from end users for Statistics Canada to research and use alternative data collection methods in CPI development.
  5. Superlative index A few of the experts and end users consulted for the CPI supported using an additional index as a complement to the CPI: a superlative, or chained, index such as the one used in the United States. The US uses a chained index but there are drawbacks from using more than one CPI including the higher costs of production and a need to effectively communicate the differences and the purpose of the additional index.Footnote 50 It was suggested that going with this approach could be an experimental index. It is believed that this could help address the issues related to substitution bias that impacts all CPIs. However, the CPI-EI is expected to make improvements in this regard by updating the CPI basket every two years rather than every four years. 
  6. Measuring housing prices — It is argued that the way housing data are produced, and the weighting of some specific data, make the current CPI a less-than-best measure for determining housing price changes.Footnote 51 Because of the importance of housing price changes, expert informants argued that an additional housing prices indicator in the CPI could be used by the Bank of Canada to make monetary policy decisions.

Question 6: Does the program demonstrate efficient and economical decision-making and implementation?

Is the CPI program duplicating or complementing existing programs/initiatives?
  • Could certain aspects or components of the program be transferred to other levels of government or other organizations (private or public sector)?

Key Findings

There was no evidence that CPI is duplicating existing programs or initiatives.

There was no evidence to suggest that the CPI is duplicating existing programs or initiatives. None of the key informants consulted could identify overlap or duplication among the CPI and similar programs and initiatives.

All those consulted indicated that the CPI should not be transferred to another organization or level of government. There are some alternative indexes, such as the Google Price Index (which is purely based on online purchases), and sub-indexes such as the Canadian Real Estate Association shelter index, but these are not seen as being as credible, comprehensive, complex or representative — nor could these replace the CPI in terms of its use.

7. Conclusion and Recommendations

7.1 Conclusion

7.1.1 Relevance

The CPI is an important measurement used by the Canadian government and other key stakeholders to inform economic and monetary policy, such as developing the GDP, measuring rates of inflation for the Bank of Canada and determining transfer payments to provinces and territories by Finance Canada. Internationally, the CPI is one of the most widely used statistics for measuring price changes and economic performance; it is the most visible and credible measure to anchor inflation expectations. Consequently, the need for the CPI continues, and the CPI remains highly relevant.

The CPI is aligned with federal and provincial government priorities. The production of the CPI is one of the ongoing priorities of Statistics Canada. It is used by all levels of government. It also responds to the priority of transparency by providing objective and non-partisan statistics.

The Government of Canada has a legitimate role and responsibility in producing a CPI, and a responsibility for it to be produced by Statistics Canada. Strong evidence shows that CPI activities are congruent with Statistics Canada's legislated role and the federal government's jurisdiction. As a member of the International Monetary Fund, the federal government must produce a CPI to meet legislative and international reporting requirements. Evaluation evidence indicates that Statistics Canada is best positioned to produce Canada's CPI, given its status as an independent agency and its expertise and capacity. This is in line with the international practice — in all jurisdictions reviewed, the CPI is produced on a monthly basis by a national independent statistical agency.

Although evidence showed that the CPI is relevant, it also revealed unmet needs, and an interest in a deeper level of data than what is now released by Statistics Canada. Consultations with end users indicated an interest in the CPD developing and issuing CPI sub-indexes — particularly in the areas of seniors and those living close to the poverty line (a basic necessities index). In fact, the need to better reflect the actual cost of living for some age cohorts was also raised in several academic publications.Footnote 52 CPD has recognized that a single CPI cannot serve all users equally well and has been examining publishing 'families' of indices.

7.1.2 Design and Implementation of CPI-EI

The CPI-EI was necessary to address known weaknesses of the CPI and ensure that the index continues to be high quality and reliable. Evidence suggests that the design of the CPI-EI is sound, and that the initiatives underway are the right ones to improve the CPI. Furthermore, evidence showed that the CPI-EI is being implemented as planned; some facets are being implemented ahead of schedule.

The CPI-EI governance arrangements are viewed as appropriate and sufficient to support the initiative and the ongoing management of associated risks. Senior CPD management showed an ability to address emerging issues and challenges by strengthening overall project management and governance, by enhancing CPD human resource capacity, and by making other organizational changes. The evaluation revealed that structures and processes are in place to monitor implementation progress of the CPI-EI, and that the initiative has contributed to building a project management capacity and culture. Some challenges related to implementation were identified, mainly: better clarification of roles and responsibilities, conveying governance arrangements more clearly to the lower levels in CPD, and more horizontal discussions of cross-cutting issues. A concern was also raised with respect to the significant subject-matter expertise lost recently in the division. Regardless of some successful capacity-building initiatives in CPD, a potential erosion of subject-matter expertise is a risk factor for the implementation of CPI-EI, particularly in relation to succession and continuity planning.

7.1.3 Performance — Achieving Expected Outcomes

Data production

Canada's CPI meets international standards and is an index that is trusted, credible, relevant and timely. Evidence also revealed that the changes ushered in under the CPI-EI are consistent with the approach found in other countries, and that the CPI-EI is needed to maintain and possibly raise confidence in the CPI. Early results of the CPI-EI show that the level of confidence in CPI data and its representation of consumer spending is being reinforced through a better quality assurance process and a more frequent and timely basket.

Dissemination and communication

Statistics Canada's CPI data, research, publications and methodology are made public in an open manner on Statistics Canada's website — even if, at times, finding some of these can be difficult. CPD appears to have well established relations with their primary clients, but have invested less time engaging broader stakeholders in the CPI and CPI-EI. The evaluation found a desire for improved communications and dissemination of more detailed information and data as well as more frequent updates. It is recognized, however, that CPD needs to balance its independence and neutrality in determining the composition of the CPI with the expectation to engage key stakeholders more broadly.

The evaluation revealed that, in terms of service quality, Statistics Canada staff is competent and have the appropriate knowledge to respond to specific demand. However, it can be difficult to identify the most appropriate expert at CPD to contact for help, or to answer CPI-related questions and issues.

Research and development

Evidence showed that the CPI data and index withstands public and expert scrutiny; however, it also showed a need for continual research and development to ensure that the CPI remains highly relevant and credible. The CPI-EI is achieving its short-term outcomes: more frequent and timely basket updates; a broader range of price quotes; a more representative sample of products and stores for sampling, including faster introduction of new products and stores to the sample, and faster reflection of changing consumer spending patterns; and advanced techniques to implement quality adjustment. The program is making progress towards enhanced design of its concepts, methods and data sources.

Information technology

Evidence indicated that, until recently, the CPI's IT and technical systems were not sufficient to meet its evolving needs. Currently, the IT enabling systems and processes introduced to support the CPI-EI are meeting CPI needs and are improving CPI data collection methods. The CPI-EI has led to general improvement of the IT infrastructure — greater efficiencies and modernization of the CPI computer systems and tools. The evaluation found some continuing challenges, including the need for better integration of IT systems for data collection, processing aggregation and dissemination. The long-term sustainability of CPI's IT infrastructure is also a concern: specifically, that in future times of resource constraints the program might not be able to maintain the implementation of its evergreen strategy for renewal of key IT systems over rolling five-year cycles.

Evidence generally indicates good progress toward the realization of CPI and CPI-EI outcomes at the immediate and intermediate level: these help achieve the final outcome, "Canadians will have accurate, timely, relevant set of consumer price indexes meant to support a variety of key policy programs." Furthermore, the relevance section demonstrated that the CPI is an excellent index when used for its intended purposes and contributes to a variety of policy programs. However, there is inherent complexity in using the same index to provide information for different uses and needs. Evaluation findings confirmed as well that the CPI-EI responds to many concerns and biases in the CPI, which will make for a more reliable and credible price index with greater internal capacity.

Performance — economy and efficiency

A review of program financial data indicates that from a financial perspective, the CPI-EI project is being implemented according to plan. The expenditures on the CPI and the CPI-EI appear to track with the original budget.

Based on the limited efficiency and economy assessment, CPI seems to use an efficiency-based approach in its practices. Some evidence showed that the CPI is efficient in its administrative and operational procedures.

There is some room to improve efficiency by implementing a performance measurement system. While no specific inefficiencies at the CPI level were revealed, some improvements in the relationships with SNA and the Survey of Household Spending were suggested to better harmonize their process. Some alternative approaches and considerations were raised and supported by the jurisdictional review, namely: moving the CPI to a one-year update cycle; making use of alternative data sources and data collection methods; examining the benefits of a superlative index and allowing for revisions; and using more seasonal adjustments.

7.2 Recommendations

Recommendations emerge from the evaluation findings that advocate strategic planning for sustainability after CPI-EI complete implementation, enhanced governance at the sub-project level, improved communication with stakeholders and accessibility of CPI products, and strengthened systematic performance measurement to demonstrate the achievement of results.

Relevance and efficiency

Recommendation 1

That CPD examines alternatives to make the program potentially more efficient and responsive to end user needs (i.e., alternative data collection sources and methods; harmonized update cycles of the Survey of Household Spending and the CPI; better alignment with System of National Accounts methodology, and additional indicators to the CPI such as housing prices, etc.).


Governance and implementation

Recommendation 2

That CPD strengthens its governance processes by:

  • improving its performance measurement system to ensure performance information is available and timely to
    • demonstrate the program's effectiveness in achieving its expected outcomes
    • support decision-making
  • ensuring there is an integrated strategic planning process in place to sustain the CPI during and after the CPI-EI covering human resources, IT and research and development
  • reinforcing governance at the sub-project level by putting in place effective mechanisms to inform and discuss cross-cutting issues at the division's lower levels.

Performance — Effectiveness

Recommendation 3

That CPI management, in partnership with Communications Division, develops a dissemination and communications strategy to ensure that up-to-date and easily accessible information is made available to the general public and to key stakeholders through:

  • up-to-date reference documents and publications
  • a user-friendly website to ensure accessibility
  • a list of official contacts available to key stakeholders
  • ongoing communications to key stakeholders, especially updates on the CPI-EI.

8. Management Response and Action Plan

Recommendation 1

Relevance and efficiency

It is recommended that CPD examine alternatives that could potentially increase the program's efficiency and be responsive to end-user needs (i.e., alternative data collection sources and methods; harmonized update cycles of SHS and CPI; better alignment with the SNA methodology, additional indicators to the CPI such as housing prices, etc.).

Statement of Agreement / Disagreement

Management agrees with this recommendation. Agency-wide solution will be implemented to address recommendations where applicable.

Management Response

The items noted above are longstanding demands from various users that are known to CPD's management.

The PMAC agreed at its October, 2011 meeting that these additional items could be given lower priority, and that CPD should focus on the top priority challenges explicitly promised by the CPI-EI: improving (1) Basket Updates, the (2) Price Sample, (3) Quality Adjustments and the (4) IT infrastructure.

However, given recent successes in year 3 of the project in making progress in each of the above, the CPI-EI Steering Committee, on November 22, 2012, agreed to expand the scope and change the name of "(3) Quality Adjustments" to "(3) Quality Improvements". This expanded project, in addition to including the mandate to improve quality adjustment methods, now includes alternate measures of shelter (including an improved NHPI and attempts to cover resale housing prices – projects being conducted by PPD with CPD support), closer alignment to SHS and SNA nomenclatures and methods, and recommendation on a standard basket classification among STC stakeholders (CPD, ISD, SNA). A project to explore and implement Alternate Data Sources (ADS) has also been added to the "(2) Price Sample" portfolio.

Table 9. Schedule
Timeline Deliverable(s) Responsible Party
September, 2013 Options and recommendations on creating a more comprehensive Residential House Price Index, introducing Condominiums in the New Housing Price Index (NHPI), upgrading the NHPI with better quality adjustment methods PPD
December, 2013 Analytical study on estimates of alternate measures of shelter in the CPI CPD
March, 2014 Final report on recommendations for ADS CPD
December, 2014 Alternate measures of shelter published monthly in CANSIM CPD
March, 2015 Recommendation on a standard basket classification among STC stakeholders (CPD, ISD, SNA) CPD

Recommendation 2

Governance and implementation

It is recommended that CPD strengthen its governance processes by:

  • Improving its performance measurement system to ensure performance information is available and timely to :
    • demonstrate the effectiveness of the program in achieving its expected outcomes
    • support on-going decision making.
  • Ensuring there is an integrated strategic planning process in place to sustain the CPI during and after the CPI-EI which includes human resources, information technology and research & development;
  • Reinforcing governance at the sub-project level by putting in place effective mechanisms to inform and discuss cross-cutting issues at the lower levels in the organization.

Statement of Agreement /Disagreement

Management agrees with these recommendations. Agency-wide solution will be implemented to address recommendations where applicable.

Management Response

Recommendation 2a)

CPD already has established key high level project indicators (e.g., frequency and timeliness of basket updates, number of price quotes to add, number of quality adjustment methods to improve) related to project outputs. However, CPD agrees that more measures of expected outcomes expressed under CPI as well as CPI-EI logic models are needed to demonstrate the effectiveness of the program in achieving its expected outcomes and will work with sub-project leaders to identify them.

Recommendation 2b)

About two-thirds of the headquarters staff (of about 150) engaged in working on the CPI are actively engaged in enhancing it, working on 6 project portfolios covering more than 50 sub-projects and 170 separate deliverables. CPD has already recognized the complexity of managing such a considerable amount of change in an environment that is cost-conscious and that will not tolerate errors or delays. It has therefore developed and very recently implemented the CPD Planning Tool, a dedicated Access database that allows managers to spell out clearly project costs, deliverables, human and IT resource needs. It also integrates all activities in the Division, including research and development and regular monthly production of the CPI. The CPD Planning Tool is greatly facilitating decision-making and has helped CPD management make key strategic decisions regarding staffing and deliverables for FY 2013-14 and 2014-15, including identifying shortfalls in some key competencies at a divisional level. It is garnering attention as a best practice at Statistics Canada.

CPD management will prepare a plan that outlines a Post CPI-EI Strategy. The CPD aims to deliver results that can reliably be maintained indefinitely after the end of the CPI-EI. Moreover, it is expected that the CPI program no longer be the recipient of LTP funds, but instead be financially "self-sufficient". To do this, the program has plans to self-fund the redesign of important systems at regular intervals, thus ensuring that they are "evergreen". The program also desires to remain flexible and evolve with changing needs and/or opportunities (e.g., incorporating alternate data sources and cutting back on price collection).

Recommendation 2c)

A well-developed project governance already exists, one that incorporates all employees internal and external to CPD. The CPI-EI Project Secretariat will work to enhance it further, notably by disseminating and sharing CPD Planning Tool reports. They will be used in various forums (such as Divisional Meetings, Project Meetings, sub-project and working group meetings) to inform staff of plans and priorities relating to projects relevant to them. Future planning cycles will also endeavor to involve staff below the Chief level more directly in the planning activity.

CPD is also planning a special one-day all-staff meeting, where the theme will be "Improving Communication". Communication around cross-cutting project issues, and how to improve it, will be addressed directly at this meeting and employee –driven suggestions considered for implementation.

Table 10. Schedule
Timeline Deliverable(s) Responsible Party
March, 2013 (Done) 2b) Delivery and implementation of CPD Planning Tool CPD
August, 2013 2c) All-staff meeting on "Improving Communication" will be held CPD
September, 2013 2c) Incorporation of CPD Planning Tool feedback in existing governance structure CPD
December, 2013 2a) Develop indicators to better measure program and project outcomes CPD
March, 2014 2b) Post CPI-EI Strategy document CPD

Recommendation 3

Performance - Effectiveness

It is recommended that CPI Management, in partnership with Communication Branch, develop a Dissemination & Communication Strategy to ensure that up-to-date and easily accessible information is made available to the general public and to key stakeholders through:

  • Up-to-date reference documents and publications;
  • User-friendly website to ensure accessibility;
  • List of official contacts available to key stakeholders; and
  • Ongoing communications to key stakeholders particularly in providing updates on the CPI-EI.

Statement of Agreement /Disagreement

Management agrees with this recommendation. Agency-wide solution will be implemented to address recommendations where applicable.

Management Response

CPI:

CPD has already recognized the need for more and better communication with its external stakeholders. To this end, it began in July, 2012 the task of updating the CPI Reference Paper, a detailed sources and methods document that had not been updated in almost 20 years. Work is advancing well, with many major changes, and it will be published in the Winter of 2014.

CPD will make available a list of official contacts to key stakeholders.

CPD also developed a report summarizing the two first years of the project. This has not yet been made available to external stakeholders, but should. A report summarizing accomplishments from the third year of the project is also planned. The report will also have a new forward-looking component that outlines plans for the coming year. This will form the basis of what will become a semi-annual report that details achievements and plans, suitable for external and internal distribution. The idea of developing a Quarterly Bulletin will also be explored.

CPD will task additional resources from its Project Secretariat, Concepts & Research and various project teams with generating more content about past, current and future CPI-EI project activities and results, and publishing it.

Table 11. Schedule
Timeline Deliverable(s) Responsible Party
September, 2013 Dissemination of Year 1-2 Report on CPI-EI Accomplishments CPD
December, 2013 Dissemination of Year 3 Report on CPI-EI Accomplishments CPD
January, 2014 List of official contacts available to key stakeholders CPD
February, 2014 Publication of revised CPI Reference Paper CPD
March, 2014 Produce semi-annual report CPD

Communication Branch Action Plan:

Communications Branch will develop a communications strategy to increase the visibility and understanding of the CPI and CPI-EI and broaden their reach with stakeholders.

The plan will make provisions for ongoing communications with the public and stakeholders through appropriate channels.

Improved presentation and accessibility of CPI information will be achieved through the deployment of Statistics Canada's New Dissemination Model.

Table 12. Schedule
Timeline Deliverable(s) Responsible Party
October, 2013 Development of CPI-EI communications strategy CPD, Communications
ongoing Generation of communications content CPD

Appendices

  • Appendix A — CPI and CPI-EI logic models
  • Appendix B — Governance and Project Management Structure
  • Appendix C — Detailed Evaluation Matrix
  • Appendix D — Summary of Key Activities and Outputs of CPI-EI
  • Appendix E — Examples of Relevant Management Practices
  • Appendix F — IMF Data Quality Assessment Framework, Canada
  • Appendix G — Literature and documents reviewed

All appendices are available upon request. Please contact AEB-Professional-Practices@statcan.gc.ca

Notes:

Footnote 1

Philippe Bergevin. "Housing Bubbles and the Consumer Price Index: A Proposal for a Better Inflation Indicator." C.D. Howe Institute Commentary no. 362. September, 2012. 20 pp.

Return to footnote 1 referrer

Footnote 2

The core CPI excludes eight of the most volatile components in the CPI: fruit, vegetables, gasoline, fuel oil, natural gas, mortgage interest, intercity transportation, and tobacco products.

Return to footnote 2 referrer

Footnote 3

Bank of Canada. The Consumer Price Index. Backgrounders. October 2013.

Return to footnote 3 referrer

Footnote 4

Ibid.

Return to footnote 4 referrer

Footnote 5

Sabourin, Patrick. 2012. "Measurement Bias in the Canadian Consumer Price Index: An Update," Bank of Canada Review, Summer 2012, pp. 1–11.

Return to footnote 5 referrer

Footnote 6

Statistics Canada. Consumer Price Index (CPI). Definitions, data source and methods.

Return to footnote 6 referrer

Footnote 7

Ibid.

Return to footnote 7 referrer

Footnote 8

Greenless, J. and McClelland, R. "Addressing misconceptions about the Consumer Price Index" in Monthly Labor Review (August 2008, pp. 3–19)

Return to footnote 8 referrer

Footnote 9

The Boskin Commission Report and Its Aftermath (NBER Working Paper No. 7759 - Robert J. Gordon (2000). Digest by Lester A. Picker

Return to footnote 9 referrer

Footnote 10

Ibid.

Return to footnote 10 referrer

Footnote 11

Statistics Canada, Refining the Blueprint to Improve the Canadian CPI. Presentation to the Price Measurement Advisory Committee, September 30, 2012.

Return to footnote 11 referrer

Footnote 12

Statistics Canada, 2011/2012 Report on Plans and Priorities.

Return to footnote 12 referrer

Footnote 13

Statistics Act (R.S.C., 1985, c. S-19)

Return to footnote 13 referrer

Footnote 14

Bretton Woods and Related Agreements Act (R.S.C., 1985, c. B-7)

Return to footnote 14 referrer

Footnote 15

The IMF's Special Data Dissemination Standard was established to guide Fund members that have access to international capital markets, in the provision of their economic and financial data to the public.

Return to footnote 15 referrer

Footnote 16

International Monetary Fund, 2007. The Special Data Dissemination System: guide for subscribers and users.
Washington, D.C.

Return to footnote 16 referrer

Footnote 17

Old Age Security Act (R.S.C., 1985, c. O-9)

Return to footnote 17 referrer

Footnote 18

Rossiter, J., "Measurement Bias In the Canadian Consumer Price Index" Bank of Canada Working Paper 2005-39.

Return to footnote 18 referrer

Footnote 19

The US Boskin Commission is recognized as the most detailed review of the CPI ever to be undertaken. Its findings and recommendations would become the benchmark for which many countries, including Canada, would use to reform their CPI.

Return to footnote 19 referrer

Footnote 20

Beatty, T. and Erling Roed Larsen, E.R., Using Engel Curves to Estimate Bias in the Canadian CPI as a Cost of Living Index, Working Paper number 2004-05, January 2004 In Food and Resource Economics, University of British Columbia

Return to footnote 20 referrer

Footnote 21

Final Report to the Senate Finance Committee from the Advisory Commission To Study The Consumer Price Index, December 4, 1996 ("The Boskin Commission")

Return to footnote 21 referrer

Footnote 22

Final Report to the Senate Finance Committee from the Advisory Commission To Study The Consumer Price Index, December 4, 1996 ("The Boskin Commission")

Return to footnote 22 referrer

Footnote 23

Ibid.

Return to footnote 23 referrer

Footnote 24

Sabourin, P., "Measurement Bias in the Canadian Consumer Price Index: An Update" in Bank of Canada Review (Summer 2012 , pp. 1-11)

Return to footnote 24 referrer

Footnote 25

Johnson, D., Reed, S. and Stewart, K. "Price measurement in the United States: a decade after the Boskin Report." In Monthly Labor Review, May 2006, pp. 10-19.

Return to footnote 25 referrer

Footnote 26

Ragan, C. Fixing Canada's CPI: A Simple and Sensible Policy Change for Minister Flaherty. C.D. Howe Institute. March 8, 2011 (3 pages e-brief)

Return to footnote 26 referrer

Footnote 27

2011-12 Progress report

Return to footnote 27 referrer

Footnote 28

'Sample design' refers to creating sampling strategies that produce representative estimates of a population. The quality of the CPI can be affected by over- or under-sampling of key products, outlets or geographies, potentially introducing bias into the estimates. Prior to the CPI-EI, sample change was primarily triggered by signals from direct field observations, and lacked the systematic basis needed to optimize the sample selection and ensure maximum representativeness. The foundation of a representative sample selection is a sound identification of the target population being studied (i.e., the key products, the outlets, geographical representation), typically called the 'frame'. An important part of the improvement of the sample design under CPI-EI is to build a sampling frame that is flexible, allowing changes to sample composition to be made in a timely manner. Statistics Canada's statistical infrastructure (the Business Registr and other surveys) are used in developing of the frame.

Return to footnote 28 referrer

Footnote 29

Governance matrix 2012-02-27.

Return to footnote 29 referrer

Footnote 30

"Governance and Management in the CPI-EI2." CPD internal document,August 2012.

Return to footnote 30 referrer

Footnote 31

As outlined in the Program Background and Objectives sections.

Return to footnote 31 referrer

Footnote 32

International Monetary Fund. October 2003. IMF Country Report No. 03/328. "Canada: Report on the Observance of Standards and Codes—Data Module, Response by the Authorities, and Detailed Assessments Using Data Quality Assessment Framework." 176 pp.

Return to footnote 32 referrer

Footnote 33

Data dimension includes criteria around prescribed coverage, periodicity and timeliness.

Return to footnote 33 referrer

Footnote 34

Access dimension relates to the advance release calendar, where subscribers (i.e., statistical agencies) must disseminate an ARC for all prescribed components of all prescribed data categories.

Return to footnote 34 referrer

Footnote 35

Integrity dimension relates to metadata certification where subscribers (i.e., statistical agencies) are required to certify, on an annual basis, the accuracy of the metadata posted on the DSBB.

Return to footnote 35 referrer

Footnote 36

Data Quality dimension refers to available information on the methodology, sources, and reconciliation of data categories in Data Quality Assessment Framework that would facilitate users to assess the quality of the data.

Return to footnote 36 referrer

Footnote 37

Users could access documents on the web earlier that fiscal year 1997/1998. However, most of them are archived in the system. The web scan conducted for this evaluation cover only Internet publications released from fiscal year 1997/1998 to 2012/2013, as of March 26, from the CPI web page.

Return to footnote 37 referrer

Footnote 38

The web scan looked at publications and data that referred to the CPI only. Particular indexes such as service price indexes and the Construction Price Index are not captured.

Return to footnote 38 referrer

Footnote 39

According to Daily 2011/2012 Webtrends Analytics Reports (from April 1, 2011 to March 31, 2012)

Return to footnote 39 referrer

Footnote 40

Data provided by Communications Division, March 26, 2013. March 2013 is not included in the total, as the release of CPI data was planned for March 27, 2013.

Return to footnote 40 referrer

Footnote 41

Microdata consist of the data directly observed or collected from a specific unit of observation (individual cases, answers from one individual, from one respondent)

Return to footnote 41 referrer

Footnote 42

2003 IMF report.

Return to footnote 42 referrer

Footnote 43

As defined by the International Monetary Fund's Data Quality Assessment Framework.

Return to footnote 43 referrer

Footnote 44

The other four aspects are covered in more detail in Section 6.1 Production on Data, and Section 6.2 Dissemination and communication of data.

Return to footnote 44 referrer

Footnote 45

Methodological soundness is defined as "[CPI is] generally following internationally accepted guidelines on concepts and definitions, scope, classification and sectorization, basis of recording, and valuation."
Accuracy and reliability isdefined by this measure: "All macroeconomic datasets get high marks for accuracy and reliability. Source data are adequate, statistical techniques are sound, and assessment and validation of source data and outputs are thorough. In several datasets, some fine tuning could be done by conducting more regular and/or thorough studies of revisions and using them to improve the statistical output."

Return to footnote 45 referrer

Footnote 46

Adapted from 2003 IMF report, p.63.

Return to footnote 46 referrer

Footnote 47

'Bricks and mortar' describes a company or portion of a company with a physical presence, as opposed to one that exists only on the Internet; 'clicks and mortar' describes a combination of traditional (brick and mortar) and online establishments, such as a store or a bank with a website. Transactions can be initiated and finalized at either the website or the physical site. Source: BusinessDictionary.com

Return to footnote 47 referrer

Footnote 48

The Consumer Price Index Enhancement Initiative 2011/2012 Retrospective Report. Consumer Prices Division, June 2012.

Return to footnote 48 referrer

Footnote 49

PMAC, Refining the Blueprint for the CPI - FINAL on 2011-09-28.

Return to footnote 49 referrer

Footnote 50

There is an ongoing debate in the literature and among economists about the proper choice among alternative estimators of price to address bias in the CPI. While it is agreed that certain superlative index will closely approximate an exact cost of living this could be done only under ideal condition and with a lag, unless correction based on the observed dispersion of price movement are done (see Diewert, 1996). It is agreed among experts consulted that a superlative index should be constructed as more up to date expenditure information is obtained and inexpensive way to correct the superlative index are developed.

Return to footnote 50 referrer

Footnote 51

More specifically, it is argued that the central bank may need a broader indicator of inflation which gives prominence to house prices rather than the cost of housing services in its conduct of monetary policy. While the actual indicator is fine for the purpose of estimating the cost of living, a broader indicator of inflation which gives prominence to house price rather than the cost of housing services in its conduct of monetary policy would be useful.

Return to footnote 51 referrer

Footnote 52

Ibid.

Return to footnote 52 referrer

Date modified: