Evaluation Report
June 2021
How the report is structured
- The report in short
- Acronyms and abbreviations
- What is covered
- What we learned
- Design and delivery
- Responsiveness to users' needs
- Moving forward
- How to improve the program
- Management response and action plan
- Appendix A: List of new economic statistical products released
The report in short
Throughout the pandemic, Statistics Canada has responded rapidly and with agility to the data needs of various users by developing many new products relevant to COVID-19 and its impacts on Canadians. While this response was agency-wide, this report focuses on three types of products: 1) gross domestic product (GDP) flash estimates, 2) consumer prices data and 3) other new economic statistical products.
This evaluation was conducted by Statistics Canada in accordance with the Treasury Board Policy on Results and Statistics Canada's Risk-based Audit and Evaluation Plan (2020/2021 to 2024/2025). The objective of the evaluation was to provide a neutral, evidence-based assessment of Statistics Canada's data response to COVID-19. The evaluation aimed at providing valuable information about the timeliness, the relevance and the impact of these products. It also looked at some of the related challenges and opportunities faced by the divisions responsible for these products to inform future direction.
The evaluation methodology consisted of a document review, administrative reviews and key interviews with Statistics Canada professionals working in different divisions within Economic Statistics. Additionally, interviews were conducted with key users and partners external to Statistics Canada. The findings outlined in this report are based on the triangulation of these data collection methods.
Key findings and recommendations
Design and delivery
The evaluation reviewed organizational changes that were implemented to develop COVID-19 economic statistical products. The evaluation found that Statistics Canada took rapid action by assessing and delivering products that shed light on issues of importance during the pandemic. Delivery of COVID-19 economic statistical products was facilitated by an organizational shift towards business agility, which included innovative approaches to manage resources and analyze data. The dedication of staff was instrumental in the successful release of those products. However, the sheer volume and push for expediency impacted staff well-being.
Responsiveness to users' needs
To inform Statistics Canada on the effectiveness of its response related to economic statistical data needs during COVID-19, the evaluation assessed the extent to which the products released addressed the needs of key users in different areas. The majority of users considered the products released to be relevant, useful, timely and accessible. Users were also satisfied with the agency's level of engagement and the methodological information released. As the pandemic has various socioeconomic impacts on Canadians, a number of gaps were identified in the areas of household economic data, employment data, and industry economic data.
Moving forward
The evaluation assessed the extent to which there is an ongoing need for COVID-19 economic statistical products and factors that must be taken into account going forward to deliver both COVID-19 products and regular activities. Users indicated an ongoing need for most of the products released, particularly for GDP flash estimates, consumer prices data and trade by exporter characteristics data. As the pandemic evolves, data on the recovery of the Canadian economy will be key for decision-making and policy development. The forward approach should focus on addressing post-pandemic data needs and areas of data gaps, while ensuring the sustainability of meeting both emerging and core economic data needs. Opportunities remain to fully explore the lessons learned from the pandemic.
Recommendation 1:
The Assistant Chief Statistician (ACS), Economic Statistics (Field 5), should ensure that a comprehensive strategy is developed to identify, prioritize and respond to emerging data needs (i.e., post-pandemic data), including areas of current data gaps.
The strategy should consider the key enablers identified, and include approaches and plans that clearly set out how:
- emerging data needs will be identified (e.g., stakeholder engagement approaches, international scan);
- current and emerging data gaps will be addressed;
- data needs will be prioritized based on intended outcomes and resource capacity; and
- ongoing monitoring of data needs will be carried out.
Recommendation 2:
The ACS, Economic Statistics (Field 5), should ensure that a lessons learned exercise from the COVID-19 pandemic is conducted to identify approaches that could be applied to respond to COVID-19 data needs and improve core activities going forward. The lessons learned should be presented to a Tier 1 committee that will provide direction on their potential implementation across the agency, as applicable. They should also be shared across the agency to foster innovation and continuous improvement in support of the agency-wide response to COVID-19.
Acronyms and abbreviations
- ACS
- Assistant Chief Statistician
- CPI
- Consumer Price Index
- GDP
- Gross domestic product
- HR
- Human resources
- IIP
- International investment position
- NSO
- National statistical office
- OECD
- Organisation for Economic Co-operation and Development
- PEAI
- Provincial economic activity index
- SEPH
- Survey of Employment, Payrolls and Hours
- TEC
- Trade by exporter characteristics
What is covered
The evaluation was conducted in accordance with the Treasury Board Policy on Results and Statistics Canada's Integrated Risk-based Audit and Evaluation Plan (2020/2021 to 2024/2025). In support of decision making, accountability and improvement, the objective of the evaluation was to provide a neutral, evidence-based assessment of Statistics Canada's data response to COVID-19.
Throughout the pandemic, Statistics Canada has responded rapidly and with agility to the data needs of various users by developing many new products relevant to COVID-19 and its impacts on Canadians. The evaluation focused on three types of products: 1) gross domestic product (GDP) flash estimates, 2) consumer prices data and 3) other new economic statistical products. The evaluation aims to provide valuable information about the timeliness, the relevance and the impact of these products. It also looked at some of the related challenges and opportunities faced by the divisions responsible for these products to inform future decisions.
Gross domestic product flash estimates
The GDP is a core economic measure of the health of the Canadian economy. Given the unique situation faced by the Canadian economy in the context of the COVID-19 pandemic and the demand for trusted information on the aggregate economic impact of this crisis, Statistics Canada has produced a flash estimate alongside the monthly official GDP release. The GDP flash estimate is an approximation of the scale of economic disruption resulting from the deliberate actions taken to protect the health of Canadians.
Consumer prices data
The Consumer Price Index (CPI) represents changes in prices as experienced by Canadian consumers. It measures price change by comparing, through time, the cost of a fixed basket of goods and services. While Statistics Canada continued to calculate the official CPI based on the fixed basket approach, it recognized that an analytical CPI series that accounts for temporary extreme shifts in consumer purchasing patterns would provide valuable insight into the impact of COVID-19 on the CPI. The derived products included adjusted price index, provincial average price tables and consumer spending patterns.
Other new economic statistical products
A number of other new economic statistical products were also developed to address specific COVID-19 data needs in the context of the Canadian economy:
- provincial economic activity index (PEAI)
- international investment position (IIP) (preliminary estimates)
- monthly trade by exporter characteristics (TEC)
- monthly transaction counts
- monthly trade of personal protective goods.
The list of specific products can be found in Appendix A.
The evaluation
The scope of the evaluation encompassed three types of products: 1) GDP flash estimates, 2) consumer prices data and 3) other new economic statistical products. The scope was established in consultation with the divisions involved in these areas.
The evaluation was conducted from November 2020 to April 2021 and covered products released from March 2020 to February 2021.
The following issues and questions were identified for review.
| Evaluation issues | Evaluation questions |
|---|---|
| Design and delivery | What actions were taken by Statistics Canada in response to COVID-19 in regard to GDP flash estimates, consumer prices and other new economic statistical products? |
| Responsiveness to users' needs | To what extent did the products released address the needs of key users?
|
| Moving forward | What factors should be taken into consideration moving forward? |
Guided by a utilization-focused evaluation approach, the following quantitative and qualitative collection methods were used.
Description - Figure 1
The figure depicts the four collection methods used for the evaluation: international scan, document review, key external interviews, and key internal interviews.
The international scan identified new products released and methods in other countries in response to COVID-19 (e.g., France, United States, Australia and the United Kingdom).
The document review included a review of StatisticsCanada's internal files and documents
The key external interviews included semi-structured interviews with: federal government departments, provincial and territorial governments, banks, academic and private sector. There were 25 external interviews conducted.
The key internal interviews included semi-structured interviews with Statistics Canada employees. There were 18 internal interviews conducted.
Three main limitations were identified, and mitigation strategies were employed.
| Limitation | Mitigation strategy |
|---|---|
| Because of the large number of users and partners using data, the perspectives gathered through external interviews may not be fully representative. | External interviewees were selected using specific criteria to maximize a strategic reach for the interviews. Different types of organizations that use COVID-19-related data extensively from a wide range of locations across Canada were selected. Evaluators were able to find consistent overall patterns. |
| Interviews have the possibility of self-reported bias, which occurs when individuals who are reporting on their own activities portray themselves in a more positive light. | By seeking information from a maximized circle of stakeholders, evaluators were able to find consistent overall patterns. |
| Limited documentation was available on the projects sampled for the evaluation. | Key staff working on the relevant projects were interviewed and a strategy to gather additional documents during the interview sessions was put in place. Additional interviews were conducted as needed to fill the gaps. |
What we learned
1. Design and delivery
Evaluation question
What actions were taken by Statistics Canada in response to COVID-19 in regard to GDP flash estimates, consumer prices data and other new economic statistical products?
Summary
The evaluation reviewed organizational changes that were implemented to develop COVID-19 economic statistical products. The evaluation found that Statistics Canada took rapid action by assessing and delivering products that shed light on issues of importance during the pandemic. Delivery of COVID-19 economic statistical products was facilitated by an organizational shift towards business agility, which included innovative approaches to manage resources and analyze data. The dedication of staff was instrumental in the successful release of those products. However, the sheer volume and push for expediency impacted staff well-being.
In response to COVID-19, Statistics Canada took rapid action to assess the impacts of the pandemic on the Canadian economy by delivering a variety of statistical products using innovative data approaches
The COVID-19 pandemic had, and still continues to have, significant impacts on the Canadian economy and society. During this crisis, the need for more timely and credible data has never been as crucial to understand and manage the economic and social effects of the pandemic. In response to COVID-19 economic data needs, Statistics Canada developed new products and adapted some of their regular products. These products aimed at providing more timely and relevant data for monitoring the impacts of the pandemic on the Canadian economy. Economic statistical products were primarily developed by the Macroeconomic Accounts Branch (Industry Accounts Division, National Economic Accounts Division and International Accounts Division) and the Economy-wide Statistics Branch (Consumer Prices Division) under the Economic Statistics Field. The Provincial Economic Activity Index was developed by the Analytical Studies and Modelling Branch (Economic Analysis Division) under the Strategic Management, Methods and Analysis Field, in collaboration with the National Economic Accounts Division.
Gross domestic product flash estimates (Industry Accounts Division)
Statistics Canada provides an advanced aggregate indicator (GDP flash estimate) of the state of the Canadian economy in addition to the monthly official GDP release. This advanced aggregate indicator is subject to future revisions as more information on economic activities becomes available over the period.
The methodology for the GDP flash estimate is based on preliminary and often incomplete information as well as other sources such as publicly released facts, including media releases and general public statements by businesses and governments. It was noted that the GDP flash estimate must be interpreted with caution, as the methodology to obtain it is different and the quality of the data is lower than that of the official estimates.
Consumer prices data (Consumer Prices Division)
Beginning in April 2020, the collection and compilation of CPI have been impacted by measures designed to limit the spread of COVID-19, as in-person field collection was conducted remotely, and imputation strategies were required to address an increased number of temporary business closures and out-of-stock products. These measures and strategies were aligned to international guidelines and followed consultation with other national statistical offices (NSOs) from Organisation of Economic Co-operation and Development (OECD) countries.
While Statistics Canada continued to calculate the official CPI based on the fixed basket approach, exploratory work recognized that an analytical CPI series that accounts for temporary extreme shifts in consumer purchasing patterns would provide additional insight into the impact of COVID-19. Using various sources of expenditure data, Statistics Canada, in partnership with the Bank of Canada, calculated adjusted consumer expenditure basket weights and an adjusted price index reflecting shifts in consumption patterns during the COVID-19 pandemic. New data tables reflecting consumption patterns during the pandemic were released. It was noted that the adjusted price index is experimental and should not be used instead of the official measure of consumer price inflation, the CPI.
Provincial economic activity index (Economic Analysis Division, Strategic Management, Methods and Analysis Field)
The experimental indexes of economic activity in the provinces and territories provide a timely summary measure of economic activity by combining existing economic data into a single index. Statistics Canada adopted a statistical model-based strategy to quickly create exploratory measures of provincial economic activity. The models look for correlations in data and typically have a different set of inputs for each province or territory, impacting the consistency and thus affecting inter-jurisdictional comparability.
International investment position (International Accounts Division)
The international investment position (IIP) is the value and composition of Canada's assets and liabilities to the rest of the world. The preliminary estimates (IIP flash estimates) for this measure were released once in April 2020 from the outset of the COVID-19 pandemic. Statistics Canada developed this flash estimate based on a methodology that focuses primarily on valuation effects. The estimate was not expected to be of the same quality as Statistics Canada's official estimates, as the model used was based on the valuation effect of 2019 and an incomplete set of surveys and administrative data sources. The resulting revisions were small; however, there were minimal fluctuations in the economy during the period covered.
Monthly trade by exporter characteristics (International Accounts Division)
The trade by exporter characteristics – Goods (TEC-Goods) program provides aggregated statistical information on the characteristics of Canadian businesses that export goods. The new data were developed based on a similar linkage methodology used for the annual TEC-Goods program. The data were seasonally adjusted, starting at the release of August 2020, to remove the effect of seasonal and calendar influences from the original data.
Monthly import transaction counts (International Accounts Division)
Statistics Canada began publishing monthly counts of import transactions in response to COVID 19 in May 2020. The counts are not seasonally adjusted and can vary from month to month based on seasonal patterns and trading day effects. In addition, they are not completely aligned with the trade values produced for the month based on the Canada Border Services Agency final accounting deadlines, as many of the transactions processed in a given month actually occurred in the previous month.
Monthly trade of personal protective goods (International Accounts Division)
Statistics Canada produced information to examine the trade in products used for the prevention, testing and treatment of COVID-19 by creating four categories: disinfectants and sterilization products, personal protective equipment, medical equipment and products, and diagnostic products. The categorization of goods was developed based on the Harmonized System classification guidelines.
Common approaches were used across OECD countries to develop products addressing COVID-19 economic statistical data needs
The United States releases advance estimates roughly 30 days after the quarter, which are similar to the early release of GDP. France has been publishing flash estimates of economic activity, including GDP, every two weeks since the end of March 2020 and has committed to do so until the end of 2021. The United Kingdom now provides early monthly estimates of GDP, in addition to their usual quarterly estimate. In terms of consumer prices data, Statistics Canada appeared to be the first NSO to develop an adjusted price index to account for the sudden shift in consumer expenditures. Similar to Statistics Canada, other countries did release information on the impacts of the pandemic. For example, Australia developed consumer demand statistical information during the pandemic using scanner data.
The PEAI is somewhat unique to Canada. Although it uses standard statistical approaches, similar to other OECD countries, its methodology is tailored to account for the various economic nuances in each province and territory. The model used was an adaptation of a United States model; however, the American activity index reports at the national level rather than the state level. New Zealand produces an index that is similar to that of the United States. The monthly TEC release is unique to Canada. Staff noted that other OECD countries showed interested in the product, particularly with regard to the disaggregation by city, country of destination, industry and other factors on a monthly basis.
Overall, the evaluation found that Statistics Canada modified its product mix more extensively than comparable NSOs to respond to COVID-19 data needs.
Delivery of COVID-19 economic statistical products was facilitated by an organizational shift towards business agility and through the agency's modernization initiatives
To ensure the timely delivery of COVID-19 products, changes were made to organizational approaches and internal processes to increase agility. The organizational shifts were also implemented to avoid interference with the delivery of mission-critical surveys, such as the regular GDP and CPI. Three main approaches were taken to that end: 1) by using different teams to develop, produce and release the COVID-19 products; 2) by developing the COVID-19 products outside the core infrastructure used for mission-critical surveys; and 3) by combining both approaches. For example, the GDP flash estimate was developed by a research and development team using their own methods and infrastructure (e.g., simpler techniques). Similarly, consumer prices data were primarily developed outside the CPI production team; however, many areas and staff were involved in the evaluation, analysis and quality assurance of the data. The work put additional pressures on all staff, which was also the case for the development of other products that could not solely rely on dedicated teams (e.g., TEC, PEAI). Other significant changes to existing processes (e.g., data production, data sources, methodology and dissemination) were also implemented to deliver other COVID-19 products (e.g., PEAI). Finally, collaboration between divisions was also enhanced and identified as a key enabler.
The evaluation did not find any major changes to governance mechanisms at the branch or division levels; regular structures were used to review and approve the COVID-19 products. However, the review and approval processes were expedited to maximize timeliness. It should also be noted that many products were already under development prior to the pandemic as part of the modernization agenda, and they were simply accelerated. For example, the Macroeconomic Accounts Branch already had discussions about the GDP flash estimate as part of the modernization agenda a few years prior to the pandemic. The pandemic accelerated its production and dissemination. The CPI scanner data, the Personal Inflation Calculator and high-capacity processing and machine learning capabilities were in use or development in the division prior to the pandemic. Some of the new products benefitted from this foundational work, which accelerated their release. Alternative data sources played a key role in the ability to shed light on shifting consumption patterns during the pandemic. This required collaboration across various divisions and fields, as well as with key stakeholders, such as the Bank of Canada. Big data methods and the use of open-source tools such as R and Python were repurposed to help develop the PEAI, and automation processes were already in place, which allowed the TEC development team to condense the time between the reference period and release date.
The collaboration and dedication of the teams were instrumental in the successful release of these products; staff well-being was impacted at times, but ongoing support was provided by management to help alleviate these impacts and support staff well-being
At the onset of the pandemic in March 2020, Statistics Canada primarily focused on maintaining its 21 mission-critical programs (e.g., GDP, CPI) to avoid any interruption of core economic measures for the proper functioning of the Canadian economy. Shortly after, other projects were rapidly launched by the agency to provide Canadians with data to better understand and manage various social and economic impacts of COVID-19.
Given the added workload and time pressure, the collaboration and dedication of teams involved in COVID-19 economic statistical products were instrumental to the timely delivery of these products. Flexible reallocation of resources also allowed for a rapid response to COVID-19 data needs, and release schedules for COVID-19 products were carefully planned with regard to core products to avoid increased pressure on teams and supporting divisions. Moreover, divisions within the Macroeconomic Accounts Branch, the Economy-wide Statistics Branch, and the Analytical Studies and Modelling Branch provided ongoing support to employees by adjusting workflows where possible to help facilitate work-life balance, holding well-being sessions and undertaking staff recognition activities. Nonetheless, the sheer volume and push for expediency had an impact on the well-being of staff.
2. Responsiveness to Users' Needs
Evaluation question
To what extent did the products released address the needs of key users?
- Relevance and Utility
- Timeliness
- Access
- Clarity (methodology)
- Engagement
Summary
To inform Statistics Canada on the effectiveness of its response related to economic statistical data needs during COVID-19, the evaluation assessed the extent to which the products released addressed the needs of key users in different areas. The majority of users considered the products released to be relevant, useful, timely and accessible. Users were also satisfied with the agency's level of engagement and the methodological information released. Because the pandemic has various socioeconomic impacts on Canadians, a number of gaps were identified in the areas of household economic data, employment data and industry economic data.
To gauge COVID-19-related data needs, Statistics Canada leveraged new and established mechanisms to engage with key users and partners; stakeholder groups were satisfied with the agency's level of engagement
The established networks between key users and the divisions responsible for regular macroeconomic and consumer price data served as the base for understanding the needs of users and how they evolved during the pandemic. Proactive environmental scans (e.g., media, requests by general users and journalists, liaison with the minister's office) provided a real-time understanding of evolving data needs. Direct consultations with various federal and provincial and territorial organizations were also used to gather needs. For example, the GDP flash estimates and consumer prices data were developed in close collaboration with the Bank of Canada. The PEAI was developed in consultation with provincial and territorial focal points. Global Affairs Canada and Export Development Canada were engaged before producing the monthly TEC data, and the monthly trade of personal protective goods data was developed in collaboration with Health Canada and other federal partners. Throughout the pandemic, continued discussions were held to gather feedback from key users and partners to further refine products and associated methodologies.
Users were satisfied with the level of engagement from Statistics Canada; many of them had an opportunity to share their COVID-19 data needs with the organization.
Economic statistical products were used for multiple purposes, with GDP flash estimates, consumer prices data and monthly TEC being the products most used
The evaluation found that the most frequently used economic products were the GDP flash estimates, consumer prices data and monthly TEC. As shown in Chart 1, these new products were used for multiple purposes depending on the type of user.
Chart 1: Purpose and use of new economical products by user groups
Federal government
- Monitor changes in the economy, analyze policy and facilitate decision-making at the deputy and ministerial level
- Determine economic impacts on different industries and supply chains
- Inform policy decisions
Provincial and territorial governments
- Compare economic impacts of the pandemic in their province relative to the rest of Canada
- Inform policy decisions
Academics and private consultants
- Develop economic forecasts
Banks and financial institutions
- Project credit losses based on economic forecasts
- Determine credit provisioning calculations
Although the other economic products covered by the evaluation were seen as relevant by key users, there were mixed views on their utility. For example, while the PEAI was deemed useful to financial institutions and federal departments, the lack of clarity on the methodology negatively impacted its use among provincial departments, academics and private consultants. The monthly trade of personal protective goods was particularly useful only at the onset of the pandemic. The international investment position and monthly transaction counts were found to be generally useful. However, the lack of disaggregated data at the provincial level decreased its relevance. Overall, the products were deemed helpful in assisting users in keeping their partners and the public informed of the impacts of the pandemic.
The products were largely considered unique by users and were their preferred source of data; some gaps were noted in the areas of household, employment and industry economic data
The products were largely considered unique and the level of duplication of information produced by other organizations was deemed low. Users across stakeholder groups indicated that they could not have accessed similar information from other sources, with a few exceptions: indexes similar to the PEAI are calculated by private-sector economists and some analysis of consumer price inflation is published by the Bank of Canada (i.e., Monetary Policy Report). However, even in cases where users could access data from other sources, users preferred Statistics Canada's data, because it is trusted, detailed and transparent, relative to other sources.
While users were generally satisfied with the products released by the organization, gaps were identified by users in the areas of household economic data, employment data and industry economic data.
Chart 2: Data gaps identified by key users regarding COVID-19 economic data
Household economic data
- Disaggregated data on the uptake of assistance programs (by demographics, sector/industry)
- Changes in the distribution of debt, savings and household disposable income
- Data on food prices and consumption for different household sizes
- More monthly indicators on household consumption spending
- Changes in the amount of sick leave and drug coverage, disaggregated by demographics
Employment data
- Increased timeliness of employment data at the industry level
- Increased timeliness of the Survey of Employment, Payrolls and Hours (SEPH) — lag is too long between release and reference period during the pandemic
- Diversity group data (e.g., government programs targeting Black Canadian businesses, women, First Nations, Asians)
- Behavioural changes that will impact employment in specific industries
- Incorporation of Canada Revenue Agency data with the SEPH and demographic data to facilitate regional-level analysis (by postal code)
Industry economic data
- Tourism and travel (domestic and international)
- Monthly provincial GDP
- Issues faced by small- and medium-sized enterprises and Canadian exporters, including a running tally of exported goods
- Impact on services sectors, particularly the food services
- Northern and rural consumer price data
The methodologies used by Statistics Canada were communicated to and understood by key users
Given that the majority of the products were developed in collaboration with key users and stakeholders, the methodologies were generally well understood. Methodological information was released through The Daily with the releases of the products. Key users indicated that Statistics Canada effectively communicated methodological information related to the products and was transparent about their limitations; the releases in The Daily were deemed to be useful in deepening their understanding about how to use the data.
Although users were generally satisfied, some users indicated that more methodological information would have been useful for the PEAI and consumer prices data. For example, users indicated that there was a lack of clarity around the methods and variables used to calculate the PEAI (e.g., the provincial indicators used). Additional information on how to interpret the data would also have been helpful. Finally, users also indicated that the methods and data sources used to calculate the adjusted price index were not clear.
The evaluation found that Statistics Canada is aware of these issues and has taken action. For example, outreach activities took place with key users to clarify methods and variables used for the PEAI. For consumer prices data, technical briefings were conducted to clarify methods and data sources.
Users were aware of releases and were satisfied overall with both the timeliness and the accessibility of the economic statistical products
The evaluation found that most key users, particularly federal, provincial and territorial users, relied on Statistics Canada staff to keep them informed of new releases, and engagement was continuous. Users were also kept informed of releases by accessing the Statistics Canada website, particularly through The Daily. To a lesser extent, social media was the mechanism through which they remain informed of releases. Two key channels to access data were direct contact with Statistics Canada and the website.
Although users were satisfied with the accessibility of those products, they made the following suggestions, which also apply to regular products:
- provide more detailed themes on the website or the portals for existing data (e.g., vulnerable groups)
- send data automatically to users who are subscribed to a mailing list
- highlight key points for each release in a brief summary so users can quickly determine if the data are relevant for them
- identify the most important issues on each Daily release.
In terms of timeliness of the products, users indicated that timely, quality data are always desirable. On the other hand, they were cognizant of the level of effort required by Statistics Canada to provide accurate detailed data that sheds light on issues of importance. In fact, a majority of users indicated that, at the onset of the pandemic, they had concerns about the capacity of Statistics Canada to meet set timelines for the release of core economic measures such as the GDP and the CPI, given the context at the time. As Statistics Canada maintained the planned releases of those regular core economic measures and later released additional COVID-19 economic statistical products, users no longer have those concerns. Key users were very satisfied with the timeliness of the COVID-19 economic products and recognized the trade-off between speed and accuracy of the information.
3. Moving Forward
Evaluation question
What factors should be taken into consideration moving forward?
Summary
The evaluation assessed the extent to which there is an ongoing need for COVID-19 economic statistical products and factors that must be taken into account going forward to deliver both COVID-19 products and regular activities. Users indicated an ongoing need for most of the products released, particularly for GDP flash estimates, consumer prices data and TEC data. As the pandemic evolves, data on the recovery of the Canadian economy will be key for decision making and policy development. Forward approaches should be focused on addressing post-pandemic data needs and areas of data gaps, while ensuring the sustainability of meeting both emerging and core economic data needs. Opportunities remain to fully explore the lessons learned from the pandemic.
While users foresee a continued need for COVID-19 economic statistical products, their focus has started to shift towards data needs to monitor economic recovery; meanwhile, efforts have been made by the agency to further improve these products
Keys users indicated that there is an ongoing need for economic statistical products going forward as the pandemic and its impacts continue. Among the products already released, they indicated an ongoing need particularly for GDP flash estimates, consumer prices data and TEC data. More generally, data tracking Canadian economic activity will be important. As the pandemic evolves and eventually ends or stabilizes, data on the recovery of the Canadian economy will be essential for decision making and policy development. In particular, data tracking business activity, labour market participation and reliance on government assistance programs have been identified as key data. However, the need for the GDP flash estimates will be reduced.
Statistics Canada is aware of the current COVID-19 data needs and is working on addressing gaps going forward. In particular, discussions were held to develop advanced GDP estimates for all provinces based on the work already done by Quebec and Ontario (i.e., release of semi-annual estimates) and to provide access to GDP inputs to users upon request. The increased timeliness of GDP data through GDP flash estimates also provides opportunities to meet Canada's international commitments towards the International Monetary Fund by providing data within targets. Consumer prices data, including the regular CPI, will benefit from a basket update in 2021, including future approaches to these updates. In addition, the use of the adjusted price index as an input will support the quality of the CPI. Discussions are also underway to provide more data on food pricing in the North and on changes in household spending. As well, research is ongoing to improve data consistency and seasonal adjustment of the PEAI to expand its use.
Forward approaches should consider post-pandemic data needs, areas of data gaps, and the ongoing delivery of both emerging and core economic products; moreover, opportunities remain to fully explore the lessons learned from the pandemic
Statistics Canada's rapid response to COVID-19 underscored its ability to meet new data needs during an unprecedented period. The COVID-19 economic statistical products released were largely considered relevant, timely and useful. Nonetheless, significant efforts were necessary to release COVID-19 products in a timely manner without compromising the ongoing delivery of core products.
With the emerging post-pandemic data needs, sustaining the ongoing delivery of both COVID-19-related products and core economic products will be a challenge. Moreover, these emerging needs will likely bring forth other areas of data gaps, in addition to those identified by key users (i.e., household economic data, employment data and industry economic data) during this evaluation.
In light of these considerations, the organization should reflect on its forward approach with a focus on identifying emerging data needs, addressing areas of data gaps where possible, and sustaining the ongoing delivery of both COVID-19-related products and core economic products. Business prioritization, human resources, information technology and source data were identified as key enablers to sustain the ongoing delivery of COVID-19 economic statistical products and core products.
Chart 3: Key enablers to sustain COVID-19 economic statistical products and core products
Business prioritization
As the pandemic evolves, users' data needs will also evolve (e.g., economic recovery), and a robust approach to prioritization that effectively balance new demands with core activities will be needed. Although the use of dedicated teams to develop COVID-19 products was seen as a key success factor at the onset of the pandemic, concerns remain about their capacity to continue delivering COVID-19 products and their regular activities, which mainly consist of research and development activities (e.g., exploring alternative data options). This poses a risk to future innovation activities in the organization.
Human resources
Human resource (HR) management will be crucial to finding, maintaining and enhancing staff expertise and skills. Timely delivery of relevant COVID-19 economic statistics was, in large part, attributable to the expertise of staff. Interviewees noted that building this expertise takes time. Teleworking will provide opportunities to expand the talent pool. However, this addresses only the more junior levels. Efforts targeting well-being and mental health of staff are also seen as contributing to organizational performance.
Information technology
The development of COVID-19 products outside the core infrastructure was important to avoid interference with core products and meet timelines. However, parallel processes pose risks, as independent development could lead to divergence. Interviewees recognized the need to eventually integrate processes into the main infrastructure and to revisit and streamline business processes and methods to deliver on new priorities more efficiently.
Source data
The use of alternative data was key in the creation of new products. The ability to further innovate and develop new products is strongly dependent on the agency's ability to gain access to key data sources as transaction data. Interviewees emphasized the need to continue exploring alternative source data going forward, including a more strategic and efficient data acquisition process. Such data would help address COVID-19 data needs and core activities.
Given that Statistics Canada is continuing to operate in an evolving pandemic, divisions involved in the COVID-19 economic products have not had the opportunity to conduct a full-fledged assessment of lessons learned aimed at finding approaches to improve core activities and benefitting the whole agency. Staff recognized the need to identify such lessons and find approaches to improve core activities to ensure that the organization as a whole builds on its pandemic experience to create synergies for the future. Nonetheless, some preliminary lessons learned were identified:
- develop a robust long-term HR plan to ensure adequate expertise and succession planning
- maintain the increased level of internal communication and collaboration implemented during the pandemic (break the silos)
- challenge the status quo to ensure continuous improvement of processes, methods and data products
- embed foresight into regular activities to proactively identify data needs.
Leveraging these lessons learned would help the organization further improve its COVID-19 data response and core activities. The lessons learned could also be shared across the agency to foster innovation and continuous improvement in support of the agency-wide response to COVID-19.
How to improve the program
Responsiveness to users' needs
Recommendation 1:
The Assistant Chief Statistician (ACS), Economic Statistics (Field 5), should ensure that a comprehensive strategy is developed to identify, prioritize and respond to emerging data needs (i.e., post-pandemic data), including areas of current data gaps.
The strategy should consider the key enablers identified, and include approaches and/or plans that clearly set out how:
- emerging data needs will be identified (e.g., stakeholder engagement approaches, international scan);
- current and emerging data gaps will be addressed;
- data needs will be prioritized based on intended outcomes and resource capacity; and
- ongoing monitoring of data needs will be carried out.
Moving Forward
Recommendation 2:
The ACS, Economic Statistics (Field 5), should ensure that a lessons learned exercise from the COVID 19 pandemic is conducted to identify approaches that could be applied to respond to COVID 19 data needs and improve core activities going forward. The lessons learned should be presented to a Tier 1 committee that will provide direction on their potential implementation across the agency, as applicable. They should also be shared across the agency to foster innovation and continuous improvement in support of the agency-wide response to COVID-19.
Management response and action plan
Recommendation 1
The Assistant Chief Statistician (ACS), Economic Statistics (Field 5), should ensure that a comprehensive strategy is developed to identify, prioritizeand respond to emerging data needs (i.e., post-pandemic data), including areas of current data gaps.
The strategy should consider the key enablers identified, and include approaches and/or plans that clearly set out how:
- emerging data needs will be identified (e.g., stakeholder engagement approaches, international scan);
- current and emerging data gaps will be addressed;
- data needs will be prioritized based on intended outcomes and resource capacity; and
- ongoing monitoring of data needs will be carried out.
Management response
Management agrees with the recommendation.
The Field will develop a comprehensive strategy to identify, prioritize and respond to emerging data needs.
This strategy will explain how data gaps will be addressed and include a process to ensure ongoing monitoring of data needs.
This strategy will consider the key enablers identified (i.e., business prioritization, human resources, information technology, source data) and form the basis for:
- Prioritization of Field resources
- Proposals for 'seed funding projects'
- External funding requests
Deliverables and timelines
The ACS, Economic Statistics (Field 5) will ensure the delivery of the Field 5 strategy document (March 31, 2022).
Recommendation 2
The ACS, Economic Statistics (Field 5), should ensure that a lessons learned exercise from the COVID-19 pandemic is conducted to identify approaches that could be applied to respond to COVID-19 data needs and improve core activities going forward. The lessons learned should be presented to a Tier 1 committee that will provide direction on their potential implementation across the agency, as applicable. They should also be shared across the agency to foster innovation and continuous improvement in support of the agency-wide response to COVID-19.
Management response
Management agrees with the recommendation.
The Field will conduct a lessons learned exercise that will enumerate:
- New data products that were developed in response to the pandemic
- New tools and techniques used to develop these products
This exercise will identify strengths and weaknesses in the approach taken, along with recommendations for improvement and best practices that could potentially be leveraged across the agency.
The results of this exercise will be presented to the Modernization Management Committee to foster a fulsome discussion on these lessons learned and their potential implementation agency-wide.
Deliverables and timelines
The ACS, Economic Statistics (Field 5) will ensure the delivery of a report on lessons learned and its presentation to the Modernization Management Committee (January 15, 2022).
Appendix A: List of new economic statistical products released
| Category | Date of release | Products | The Daily release |
|---|---|---|---|
| Gross domestic product by industry: Preliminary Estimate / Flash Estimate | April 15, 2020 (March flash GDP) | Statistics Canada has been producing flash estimates of monthly GDP starting with the March 2020 reference period and released the product in The Daily. | Gross domestic product by industry: Nowcast, March 2020 (March flash GDP)Table note * |
| Gross domestic product by industry: Preliminary Estimate / Flash Estimate | May 29, 2020 (April flash GDP) | Statistics Canada has been producing flash estimates of monthly GDP starting with the March 2020 reference period and released the product in The Daily. | Gross domestic product by industry, March 2020 (April flash GDP)Table note * |
|
|||
| Category | Date of release | Products | The Daily release |
|---|---|---|---|
| Monthly trade of personal protective goods | June 4, 2020 | Trade in medical and protective goods, April 2020 (first release) |
Canadian international merchandise trade, April 2020 Note: Releases occurred every subsequent month. |
| International Investment Position - Preliminary estimates | April 27, 2020 (one-time) | Estimated quarterly changes in Canada's international investment position, first quarter 2020 | Market turbulences and COVID-19 outbreak: The impact on Canada's international investment position: Preliminary estimates, first quarter 2020 |
| Monthly trade in goods by exporter characteristics | May 25, 2020 |
Trade in Goods by Exporter and Importer Characteristics: Interactive Tool Note: Monthly trade in goods by exporter characteristics data are available upon request. |
Monthly trade in goods by exporter characteristics, March 2020 (first release) Note: Releases occurred every subsequent month. |
| Monthly transaction counts | May 13, 2020 | Data: Monthly import transaction counts (not seasonally adjusted) |
Monthly trade in goods by exporter characteristics, March 2020 (first release) Note: Releases occurred every subsequent month. |
| Monthly transaction counts | May 13, 2020 | Analysis: Recent Developments in the Canadian Economy, 2020: COVID-19, first edition |
Canadian merchandise import transaction counts, April 2020 Note: Releases occurred every subsequent month. |