Chapter 3.4: Data collection planning and management

1. Collection governance

There are many governance approaches to organizing collection operations. The ultimate indicator for the adequacy of the chosen governance structure and mechanisms, however, is how well they satisfy the following strategic goals:

• Effective planning and coordination of capacity and capabilities across the collection infrastructure;
• Continuous improvement of efficiency, quality and timeliness;
• Effective reduction of response burden and costs; and
• Modernization of collection tools and processes with minimal or acceptable levels of risk.

The most appropriate governance structure is probably one that provides an optimal balance of centralization and decentralization in the statistical system context. Decentralized collection activities at the regional level or by statistical domain can sometimes lead to better quality and, to some extent, reduced costs because local knowledge can be used to improve respondent relations and increase efficiency by reducing the cost of follow-up of incomplete questionnaires, refusals, other non-response, etc. On the other hand, a centralized structure tends to facilitate planning and coordination while reducing duplication of effort. It also facilitates the implementation of consistent and standard concepts and procedures across regions and survey programs, which has an ultimate impact on data quality.

At Statistics Canada, all collection activities have been centralized under the Collection and Regional Services Branch (CRSB). Creating this centre of expertise for collection planning, development, and research, and for collection management was one of the first priorities of the Corporate Business Architecture (see Chapter 3.1: Corporate Business Architecture). Under this modernization initiative, all collection activities that were being conducted in various subject-matter areas were integrated under the CRSB, allowing Statistics Canada to find improvements and savings in its collection process, strengthen the links between all collection partners, enhance respondent relations and facilitate the modernization of collection processes and systems regardless of mode.

The CRSB includes the following units:

• The Collection Planning and Research Division (CPRD—at head office), is the centre of expertise for survey planning, survey development, and survey-collection research. CPRD's responsibilities include (1) the planning of capacity and capabilities across the collection infrastructure; (2) the delivery of coordinated collection services for household and business surveys (initial consultations, feasibility assessments, and cost estimates); (3) the development of collection strategies for all collection methods, including electronic questionnaires (EQ), computer-assisted telephone interviewing (CATI); computer-assisted personal interviewing (CAPI); paper and pencil interviewing (PAPI); and multi-mode collection; (4) the testing and support of survey-collection applications and IT-related infrastructures, and; (5) survey-collection research to improve the quality, timeliness and cost of survey collection as well as identifying ways to reduce respondent burden.
• Regional offices coordinate all modes of survey collection activities, in accordance with negotiated budgets and survey requirements. Regional offices are responsible for day-to-day management of collection operations, respondent relations, and the identification and resolution of problems related to collection as they arise.
• Statistical Survey Operations (SSO) is a separate employer managed by the staff of the Regional Offices. This organization collects information from survey respondents on their business, household or institution on behalf of Statistics Canada. SSO interviewers play a crucial role in explaining the importance of Statistics Canada surveys and engaging respondents to participate in them. Approximately 2,000 employees work for SSO and their workload depends on the survey collection volume and schedule. SSO has one employee classification group, which consists of two levels: interviewers and senior interviewers. There are also two types of interviewers: (1) field interviewers who carry out survey activities, in person, outside Statistics Canada offices, known as CAPI, and (2) interviewers who conduct interviews by telephone, known as CATI, and who work primarily from Statistics Canada regional offices..

In terms of coordination within the overall organizational governance, the CRSB seeks advice and guidance from the Agency's Collection Planning Committee. The mandate of this committee is to review and recommend strategies for generating efficiencies in collection, respondent relations, and operations in terms of cost, quality and timeliness. The primary objectives are to

• review and recommend approaches to optimize Statistics Canada's collection services in terms of methods, operations, cost-effectiveness, and minimizing of response burden;
• review and recommend actions or best practices to optimize response rates, data quality, and respondent outreach in support of data collection activities;
• review demand and priorities for collection services and to recommend necessary capacity adjustments, and
• review and recommend internal rates for collection services to Executive Management Board.

The committee also develops an annual work program and reports progress directly to the Executive Management Board, with a view to ensuring coherence between all management strategies and practices within the organization.

2. Collection planning

A good collection planning system should allow NSOs to effectively prioritize collection activities, coordinate capacity and capabilities across the collection infrastructure, and establish roles and responsibilities regarding all aspects linked to collection, including execution, assessment, monitoring, contingency planning and security, as well as the related communications strategy. The requirements of the subject-matter divisions—and the means by which, and the extent to which, these requirements are expected to be met—are examined and agreed on. It is during this phase that the resources, the funding requirements, and the schedule of activities are developed. The quality of the planning phase is crucial to the quality of the entire project. Good planning requires good management skills and knowledgeable and experienced people.

The following are key considerations in planning survey collection activities:

• Survey objectives (information needs, primary uses and users of the data, concepts, survey content, quality requirements by domains of interest and analysis plan);
• Survey frame (target population, area vs. list frames, use of existing frame vs. cost to build new one and quality of frame, including contact information);
• Sample design (statistical units, sample size, sample distribution);
• Questionnaire design (i.e., questionnaire content, types of questions (closed vs. open answers), question sensitivity, order of questions, average length of questionnaire);
• Collection method (i.e., self-enumerated vs. interview-assisted, face-to-face vs. telephone interviews, paper-and-pencil, computer-assisted or electronic questionnaires, use of electronic digital assistance devices.). Choosing the appropriate collection method is crucial and should take into account the participation rate, respondent burden, and budgetary and operational constraints. Refer to box 3.6.1 (at the end of this chapter), which identifies the collection methods used by statistical organizations and their advantages and disadvantages with regard to the entire data collection and production process;
• General consideration related to collection management (staff availability; labour market; remoteness of the target population; respondent cooperation;; transportation expense costs; average costs related to the printing or design of collection tools; language of respondents and translation costs, etc.).

Once the key planning decisions are taken, the project schedule must be created. The creation of the project schedule consists in preparing the list of activities and tasks along with timelines, key milestones, and the name(s) of the person(s) responsible for completing them. Steps include integrating lessons from the last collection exercise through a review of the post-mortem document and a peer review of the new survey procedures by collection committee. Steps also include an analysis of the interviewer capacity to determine the following: where the work will be allocated; which Management Information System reports should be used to monitor collection, and determine the parameters, conditions, and key performance indicators for identifying critical collection milestones. Finally a Collection and Operations Service Agreement is drafted and signed off by all service areas involved in all stages of the process, from survey development to collection.

At Statistics Canada, planning has been greatly improved by the creation of a single organizational point of entry for all subject-matter divisions requiring collection services—the Collection Front Door. This service is responsible for the initial collection feasibility assessment stage, during which survey specifications are reviewed, data-collection process flows are identified, required capacity is assessed, and preliminary cost estimates are prepared. Having collection experts assisting survey managers in the design of their surveys and the establishment of collection-related costs improve quality and efficiency with regard to the general planning and the conduct of surveys.

3. Collection management practices and tools

Within the collection structure, interview-based collection activities are usually performed through a direct hierarchy involving collection project managers, supervisors and interviewers. Project managers play a liaison role with the survey programs and ensure that collection requirements, such as quality standards and response rates, are met. Collection schedules, timetables for data collection, details of survey milestones, targeted response rates, and reporting periods can facilitate their work.

Supervisors are usually responsible for hiring, training and monitoring interviewers; distributing the workload among the available resources; and addressing day-to-day management problems. Supervisors' manuals assist them in performing their duties. These useful tools typically include guidelines on the following: hiring and training interviewers; creating interviewer assignments; occupational health and safety; quality and performance controls; logistics, security and privacy protection; conversion of refusals to respondents; and methods for dealing with disabled persons, language problems, or special considerations applicable to respondents.

Interviewers can be responsible for several of the following functions: listing, interviewing and tracing respondents; following-up with non-respondents; editing and coding. Although this type of work does not require advanced education, interviewers' strategic role in data production requires minimum abilities and personal qualities. Criteria to consider when hiring interviewers include interpersonal skills, fluency in local languages, organizational skills, computer literacy, trustworthiness, subject-matter knowledge, and knowledge of the area where the survey is being conducted. Interviewers' training should focus on (1) the development of soft skills, such as empathy, active listening, refusal conversion; (2) the understanding of survey concepts, content and particularities; and (3) the collection process, including the use of listing, collection and coding tools. Cognitive studies show that classroom training combined with home study, mock interviews, and early feedback on performance tend to produce better results. Regular refreshers and interviewers' manuals can also facilitate the retention of the information provided during training.

Self-administrated surveys do not require the same hierarchy but may nevertheless require listing, coding, editing, and non-response follow-up, which can be centrally managed or not. Proper training, supporting documentation, and monitoring are therefore necessary.

Recent improvements in conducting survey collection were implemented further to the analysis of collection process data. Paradata research has been focused on improving the understanding of data collection processes and response patterns leading to the identification of strategic improvement opportunities. For example, at Statistics Canada, research findings on paradata produced from CATI surveys have indicated that the same data collection approach does not work effectively throughout an entire data collection cycle. There is therefore a need to develop a more flexible, responsive and efficient data collection strategy.

One strategy is to offer a multi-modal collection option to all respondents. Under this approach, a survey collection could start in one mode and be pursued using any other combination of modes, with a view to achieving the best possible quality in the most efficient way. For example, the approach used in the Canadian census is one that starts collection with the least expensive mode (electronic questionnaire) and ends with the more expensive mode that typically yields higher response rates (personal interview). To enable this, the statistical organization needs to be able to securely and easily move cases between its electronic questionnaire platforms, its multiple call centres, and its workforce of personal interviewers. Calls must be made at times when they are most likely to generate a response. As well, information about individual cases must be available in real time, so that the organization can detect issues related to interviewer or questionnaire performance. All of this work must be accomplished with the smallest possible number of systems and processes to achieve economies of scale.

Statistics Canada has started to implement this vision in 2011, through the Integrated Collection and Operation Systems (ICOS) project. This initiative's objective is to develop an integrated collection systems environment to achieve the targeted level of flexibility between modes and sites, fully exploiting use of the Internet for electronic questionnaires. In 2014, through Phase 1 of the ICOS Business Collection Portal (ICOS-BCP), data collection started to make use of a single electronic questionnaire for both respondents and interviewers to collect data for the annual surveys of the Integrated Business Survey Program. Over the next few years, the migration of business surveys to the ICOS-BCP will continue as they align with the adoption of common data processing tools and the implementation of the use of a single electronic questionnaire to support both self-response and interviewer-administered response.

The integrated collection system environment also relies on the use of the ICOS Collection Management Portal (ICOS-CMP) to meet the needs of the census programs, social surveys, and some business surveys requiring a CAPI component. The 2016 Census of Population and Census of Agriculture will be conducted through this portal, which will also support the recruitment of Statistics Act census employees. In parallel, work will continue on addressing the business requirements of other survey programs relying on the ICOS-CMP for their data collection. In particular during this period, a prototype of an off-line solution will be implemented to allow data collection for the consumer price index program and personal interviewing for social surveys. This work will position the organization for the introduction of live data collection for surveys by means of the ICOS-CMP, beginning in 2017/2018.

4. Collection monitoring

Collection monitoring should be considered as a strategic tool to improve data quality and efficiency in collection operations. The adoption of computer-based collection technologies helps ensure (1) that the questionnaire is being used properly, (2) that the interviewing techniques are effective and consistent across interviewers, and (3) that respondents' answers adequately reflect what is intended to be measured. For example, CATI technologies often allow supervisors to listen in on live interviews. Some CAPI systems can now record parts of interviews for quality assurance purposes. Electronic questionnaires can also be programmed to prompt additional verification questions through built-in edits. Finally, personal digital assistants (PDAs) or tablets equipped with a global positioning system have also been used to verify that interviewers actually collected their information from the expected location. This technology is not yet available at Statistics Canada; however, the agency is actually exploring its possible use and feasibility for the Consumer Price Index.

More recently, the relationship between the quality, cost, productivity, and responding potential of outstanding cases over the course of collection has been investigated. Additional tools have been developed to better assess and monitor progress, quality and performance during collection and to allow the development and implementation of a Responsive Collection Design (RCD) strategy for CATI surveys. RCD strategy is an approach to survey data collection that uses both information available before and paradata accumulated during collection (e.g., the sequence of calls) to identify when changes to collection approaches are needed in response to the progress of collection. The main idea is to constantly assess the data collection process on the basis of the most recent information available with a view to making the most efficient use of available resources remaining (adaptive collection).

In practice, the RCD approach monitors and analyzes collection progress against a pre-determined set of indicators for two purposes: to identify critical data collection milestones that require significant changes to the collection approach and to adjust collection strategies to make the most efficient use of remaining available resources. This type of monitoring is central to the Integrated Business Survey Program's approach adopted by Statistics Canada (see Chapter 3.3: Enhancing how surveys are conducted).