Modelling March and July Ending Stocks Using Survey and Administrative Data

1. Introduction

This report provides the background, general methods and results of the model used to produce estimates for on-stocks of principal field crops in March and July. The work was completed by the Agriculture Division and the Economic Statistics Methods Division at Statistics Canada.

The general methodology for the model – which lists the targeted crop-province combinations - is outlined in Section 2. In Section 3, the data sources used to build the predictor set are listed. Section 4 contains the modelling methods and evaluation metrics used, while the fifth section outlines the results.

2. General methodology for ending stock modelling

A methodology for modelling on-farm stocks of principal field crops was developed and tested on the crop-province combinations that are typically published from the Field Crop Reporting Series, as shown in Table 1. These crop-province combinations account for nearly the entirety of field crops stored on Canadian farms.

The goal of the model is to produce official and accurate estimates of March and July ending on-farm stocks for select crop-province combinations using information from existing data sources. The quality of model estimates will be discussed further in sub-section 4.3 and Section 6.

3. Data sources used in the model

The modelling methodology used three data sources: 1) Statistics Canada's Field Crop Reporting Series data, 2) Statistics Canada's Farm Product Price Index and 3) alternative data from the Canadian Grain Commission.

3.1 Crop Reporting Series data

Statistics Canada's Field Crop Reporting Series obtains information on the Canadian grain industry (for details regarding the methodology of the Field Crop Reporting Series, please visit Field Crop Reporting Series). The surveys collecting on-farm stocks have historically been run each year in March (cancelled in 2023), July (June, starting in 2020 onward), September (cancelled in 2016), November, and December. As the crop year progresses, different aspects of the grain industry are collected. Some of these measures were used directly (e.g., seeding intentions, on-farm ending stocks, the production of field crops), and some of them were used to derive new predictors (e.g., the percentage of supplies which were produced, the percentage of disposition which was delivered).

3.2 Farm Product Price Index data

The Farm Product Price Index publishes estimates on the price of many different agricultural products, including nearly every grain and field crop for which ending stocks are published. Prices are published on a monthly basis for each province. Since the analysis spans more than 15 years, inflation would obfuscate the relationship between price and ending stocks. Therefore, price in its raw form was not a suitable predictor. Raw monthly provincial prices for each grain and field crops were transformed into two predictors (year over year percentage difference, quarter over year percentage difference) which better suited the model.

3.3 Alternative data from the Canadian Grain Commission

The Canadian Grain Commission (CGC) collects data concerning many aspects of the grain industry in Canada. Among these are field crops deliveries from farms to elevators. Deliveries are disseminated each week as part of the CGC's Grain Stats Weekly program at the provincial level.

Given the release frequency of CGC data, five predictors were derived: total deliveries and total feed deliveries during both the December-March and March-July Field Crop Reporting Series survey periods, alongside cumulative feed deliveries since the start of the crop year.

3.4 Larger predictor pool in July relative to March

Because July model inputs exhibit higher variability than those in March, the July model requires additional predictors to achieve strong predictive power. Testing confirmed that these extra predictors did not improve the March model's performance; therefore, they were excluded from the March model to avoid unnecessary complexity. These different predictor pools were found to produce the optimal results for each model.

While the March model uses the percentage of leftover supply (ending stocks plus feed, waste, and dockage) that was ending stocks during the previous survey period, the July model uses that value for the previous three survey periods.

The models also differ in their feed delivery predictors. Both models include total feed deliveries since the previous survey period. The July model expands on this by incorporating the proportion of deliveries that are feed deliveries over that same time frame as well as two lagged predictors: total feed deliveries and the proportion of deliveries that are feed deliveries from two survey periods prior. In contrast, the March model includes total feed deliveries and the proportion of deliveries that are feed deliveries since the beginning of the crop year.

4. Modelling March and July ending stocks

4.1 Modelling methods

The model is constructed using the historical relationships between the published estimate of on-farm stocks of grain and various characteristics of the crops and grain industries. Data from 2006 onward are used in deriving the model.

The on-farm stocks of grains are estimated using the statistical programming language R. Two learning algorithms are employed: Elastic Net from the glmnet package for feature selection and robust regression from the MASS package for prediction.

The Elastic Net procedure selected optimal model parameters using leave-one-out cross-validation to balance model fit and predictive performance. Predictors which haven't been shrunk to zero from the Elastic Net are passed into the robust regression. The robust regression uses the MM-estimator.

4.2 Defining the Target

The highest quality estimate of on-farm stocks of grains is the published estimate. Historically, the published estimate was derived from the Field Crop Reporting Series survey estimate which had been adjusted using the Supply and Disposition equation and alternative data sources as reference. This process is outlined in greater detail here: Supply and Disposition of Grains in Canada.

The supply and disposition equation is:

$\mathrm{DISPOSITION}=\mathrm{DELIVERIES}+\mathrm{SEED}+\mathrm{ENDING STOCK}+\mathrm{FEED, WASTE, DOCKAGE}$

and

$\mathrm{SUPPLY}=\mathrm{BEGINNING STOCKS}+\mathrm{PRODUCTION}$

where

$\mathrm{SUPPLY}=\mathrm{DISPOSITION}$

$\mathrm{DELIVERIES}$ estimates are taken from administrative data from the CGC.

$\mathrm{PRODUCTION}$ estimates are taken from the November Field Crop Survey for the March and July models.

$\mathrm{BEGINNING STOCKS}$ are taken as the ending stocks of the December Field Crop Survey and the March Field Crop Survey for the March and July models, respectively.

$\mathrm{ENDING STOCKS}$ estimates had been derived from the March Field Crop Survey and the July Field Crop Survey for the March and July models, respectively.

$\mathrm{SEED USE}$ estimates are derived from the November Field Crop Survey and the March Field Crop Survey, for the March and July models, respectively.

$\mathrm{FEED}\text{,}\mathrm{WASTE}\text{and}\mathrm{DOCKAGE}$ estimates are the residual of the equation outlined above.

When it comes time to estimate on-farm stocks of grain, the estimates for all variables but feed, waste and dockage are available. Therefore, the equation can be re-written as:

$\mathrm{DISPOSITION}-\mathrm{DELIVERIES}-\mathrm{SEED}=\mathrm{ENDING STOCK}+\mathrm{FEED,WASTE,DOCKAGE}$

or

$\mathrm{LEFTOVER}=\mathrm{ENDING STOCK}+\mathrm{FEED,WASTE,DOCKAGE}$

The model is ultimately built to replace the March and July on-farm ending stocks estimates, however, given the leftover amount is known, it is used to stabilize the target. The model target is the percentage of leftover which are ending stocks.

Like the survey estimate, the modelled estimate is subject to adjustments using the Supply and Disposition equation and alternative data sources as reference.

4.3 Model evaluation methods

During the research phase of this project, many model parameters were considered. This included the learning algorithms used for prediction, the size of the predictor pool, and the shape of the target.

Evaluating the success of the model was done by measuring the accuracy and bias of different parameter selections by comparing the predictions to the published March and July ending on-farm stock estimates at both the crop and province-crop levels. In both cases, the weighted mean absolute percent error (wMAPE) was used as the accuracy metric and the relative mean error (ME) as the bias metric. Assessing both metrics ensures predictions are not only close to the actual values on average, but also free from systematic directional errors that could distort estimates.

The formula for the weighted mean absolute percent error is below:

$\mathit{wMAPE}=\frac{\sum _{i=1}^n\mathit{|}{\mathit{published}}_i\mathit{-}{\mathit{prediction}}_i\mathit{|}}{\sum _{i=1}^n{\mathit{published}}_i}$

The formula for the relative mean error is below:

$\mathit{Relative ME}=\frac{\sum _{i=1}^n\left({\mathit{published}}_i,-,{\mathit{prediction}}_i\right)}{\sum _{i=1}^n{\mathit{published}}_i}$

where $\mathit{prediction}$ is the model prediction, $\mathit{published}$ is the published estimate, $\mathit{i}$ is a given observation and $\mathit{n}$ is the total number of observations in a crop or province-crop group.

5. Results

Overall, the model compares favourably to the results of the March and July surveys.

For the March period, in all but two crops – barley and corn – the model outperforms the survey. The survey only minimally outperforms the model in barley and corn. Further analysis for barley was conducted (feed deliveries published by the CGC were explored, subject matter experts were consulted), but nothing adequately explained the discrepancy.

For the July period, in all crops the model outperforms the survey. The improvements are even more significant than those in the March period. The bias distribution across all crops sees a drastic improvement in March and July, becoming more tightly clustered and centered around zero.

6. Data Quality Indicators

6.1 Confidence Interval (CI)

All predictive models are subject to errors; therefore, it is important to measure the degree of uncertainty in model estimates. This is done with the use of the confidence interval.

The following is a concise guide to the bootstrap estimation of variance for the March and July ending stocks models. The main goal of this estimate is to capture the variance of the entire modelling process. Consequently, the key element of the bootstrap – sampling with replacement – occurs at the beginning of the modelling process.

Step 1: Stratified Random Sampling with Replacement

The training set, which contains all the observations the original model is built from, is stratified by province and then randomly sampled with replacement at the observation level. If there were 10 observations initially, then the new bootstrap set will also have 10 observations, but there may be recurrences of the same observation due to the stratified random sampling with replacement.

Step 2: Elastic Net Regression to find Optimal Penalty Size

An Elastic Net regression is cross-validated (at 4 folds, trying 100 different penalty sizes) on the bootstrap modelling set to find the optimal penalty size. This optimal penalty size is then used in the next step.

Step 3: Elastic Net Regression to Create Reduced Pool of Predictors

The Elastic Net is retrained, using the optimal penalty size derived in Step 2. The resulting optimal model may shrink some predictor coefficients to zero. In this case, the predictors would be removed from the initial pool of predictors, creating a reduced pool of predictors.

Step 4: Robust Regression with Reduced Pool of Predictors

The bootstrap modelling set is cut down to the reduced pool of predictors and then fitted with a robust regression. This robust regression is the model which provides the final estimate.

Step 5: Prediction is Made using the Robust Regression Model

The trained robust regression is applied to the prediction set.

Step 6: Repeat Steps 1-5 1000 times

Steps 1-5 are repeated 1000 times, with each new stratified random sample providing a different mutation of the initial modelling set.

Step 7: Calculate CI using Percentile Method

After the 1000 predictions have been saved, the final output is created like this:

• Order the 1000 replicates
• Save values of 2.5 and 97.5 percentiles (25^th and 975^th largest replicates).

Step 8: Repeat Steps 1-7 for each crop

All previous steps must be repeated for each crop.

We can build a confidence interval using the 1000 bootstrap replication. We use the percentile method again and remove the bottom 25 and top 25 bootstrap replicates, which gives us the 95% confidence interval.

These intervals can be used by Subject Matter in their validation process and during the Supply and Disposition process.

6.2 Robust Coefficient of Variation (CV)

Consistent with our confidence interval approach, we use a robust measure of variability to reduce the influence of extreme bootstrap estimates, which may disproportionately inflate estimated variability for province-crop combinations with small sample sizes. The robust coefficient of variation (CV) for each province-crop is defined as:

$\mathit{Robust CV}=\frac{\mathit{MAD(bootstrap estimates)}*{\displaystyle \frac{1}{{\Phi }^{-1}\mathrm{(0.75)}}}}{\mathit{prediction}}$

where, $\mathit{MAD()}$ denotes the median absolute deviation of the bootstrap estimates from their median, ${\Phi }^{-1}\mathrm{(0.75)}$ is the third quartile of the standard normal distribution, and $\mathit{prediction}$ is the estimate obtained from the robust regression. The product in the numerator yields a consistent estimator of the standard deviation under the assumption of an approximately normal bootstrap distribution. Dividing this quantity by the predicted estimate produces a robust coefficient of variation.

7. Release Criteria

A set of rules based on the robust coefficients of variation for each province-crop were established to determine which modelled ending stocks are of an acceptable level of quality to publish. Based on these rules, estimates that do not meet quality standards may not be published.