By: Stan Hatko, Statistics Canada

A modernization effort is underway at Statistics Canada to replace agricultural surveys with more innovative data collection methods. A key part of this modernization is the use of remote sensing classification methods of land use mapping and building detection from satellite imagery.

Currently, Statistics Canada conducts the Census of Agriculture every five years to collect information on topics such as population, yields, technology and agricultural greenhouse use in Canada. Data scientists have been teaming up with subject matter experts to modernize the collection of these data, as traditional methods are not sustainable in the long term. Innovative methods are needed to ensure the agency can continue to produce new information for the agriculture sector in an efficient manner. This project will allow the agency to make data available in a more timely manner and reduce the response burden for agricultural operators.

This project explores the machine learning techniques used to detect the total area of greenhouses in Canada from satellite imagery.

Satellite imagery

This project used RapidEye satellite images which have 5-metre pixel resolution (that is, each pixel is a 5 m by 5 m square) with 5 spectral bands.

This imagery was chosen due to its relative availability and cost. Lower resolution imagery is not always adequate to detect greenhouses, and higher resolution imagery would have proven prohibitively expensive given the total area required to cover the Canadian agricultural sector.

Labelled shape data

For certain sites the subject matter experts labelled data in the form of Shapefiles indicating which areas correspond to greenhouses. This was done manually by looking at extremely high resolution satellite and aerial imagery (using Google Earth Pro and similar software), and highlighting the area corresponding to greenhouses.

These labelled data had two roles:

• Training data (from certain sites) to build a machine learning classifier to determine the area covered by greenhouses.
• Testing data (from other sites) to evaluate the performance of the classifier.

Labelled data from Leamington, Ontario; Niagara, Ontario; and Fraser Valley, British Columbia were produced. Certain sites were chosen as training sites (like Leamington West), while others were chosen to be testing sites (like Leamington East).

Figure 2 is an example of RapidEye imagery of a region together with the greenhouse labelling file.

.

The labelled data were broken down into sites and sub-sites to train and validate the machine learning model. The training sites were:

• Leamington West
• Niagara North: N1, N1a, N3
• Fraser South: S1, S2, S3, S4, S5

Validation sites were used to test the model were:

• Leamington East
• Niagara South: S1, S2
• Fraser North: N2, N3, N5

Machine learning methodology

For each point, the data scientists needed to predict if it corresponded to a greenhouse or not, as well as a predicted probability of each point being a greenhouse.

For prediction, given a point, a window of specified size was taken around the point. This was fed in the data in this window to the classifier, which attempted to then predict if the central point is a greenhouse or not. The window around the point provided additional context to help the classifier determine if the central point is a greenhouse or not.

This process was repeated for every point in the image (except near borders), and resulted in a map showing the exact area covered by greenhouses.

For training, a sample of many such points (with the window around each point) was taken and fed in (with the label) to construct the model. The training set size was also increased by applying various transformations, for instance rotating the input image by different angles for different points.

Initial work and transition to cloud

Originally the work was done on a Statistics Canada internal system with 8 CPU cores and 16 GB of RAM. Several algorithms were tested for the classifier, including support vector machines, random forests, multilayer perceptron, and multilayer perceptron with principal component analysis (PCA).

The best results were obtained with PCA and multilayer perceptron, resulting in an F1 Score of 0.89 to 0.90 for Leamington East. Various system limitations were reached during this work, such as a lack of a dedicated Graphics Processing Unit (GPU). The GPU is required to efficiently train more complex models involving convolutional neural networks.

The public cloud was explored as an option as there were no sensitive data for this project. The project was transferred to the Microsoft Azure cloud, on a system with 112 GB of RAM, large amounts of storage and a very powerful NVIDIA V100 GPU. The Microsoft Azure Storage Explorer software was used to transfer data to and from the storage account.

Convolutional neural networks

Convolutional neural networks (ConvNets) incorporate the concepts of locality (neighbourhood around point in image being important) and translation invariance (same features useful everywhere) into a neural network. Architectures based on this have been considered state-of-the-art in image recognition for several years.

A layer in a basic ConvNet works as follows:

• Around each point in the image or previous layer, a small window (for instance, 3x3) is taken.
• The data in that window are multiplied by a matrix, to which the activation is applied (a bias can be added as well).
• This process is repeated for every point in the image (or previous layer), to obtain the new layer. The same matrix is used each time.

This equivalently corresponds to multiplying by a large sparse matrix, with certain weights tied to same values, followed by the activation.

Many different architectures based on ConvNets are possible. This project tested the following options:

• Simple ConvNet: Apply convolutional layers in sequence (output of layer is input to next), followed by fully connected.
• ResNet: Apply convolutional layer with same size output, and add to original (so input of next layer is sum of original and this layer). Can repeat this for many layers. Has been used to train extremely deep networks.
• DenseNet: Apply convolutional layer, and append outputs to original as new channels. Each layer adds new channels, which can be useful features.
• Custom branched architecture: Crop central part of window, and apply one convolutional network. Take the whole image, and apply another network (with more dimensionality reduction based on pooling layers). Merge both at the end in fully connected layer. This allows the user to focus in on the part near the central point, while getting some context around it.

The data scientists used the custom branched architecture for this project, as shown in figure 5.

For optimization, the ADAM optimizer was used with a learning rate of 10^-5. A mini-batch size of 5,000 was used, and the training was done for 50 epochs.

Results

After the model was trained, it was tested on each of the validation sites in Leamington East, Niagara South, and Fraser North. The results are summarized in the table below.

For Leamington, the result obtained was very good: the greenhouses were picked up well and false positives were small. The number of misclassified points (FP and FN) was much smaller than both the correct classes (TN and TP). This area has the best overall F1 score at slightly over 0.94.

For Niagara, the results were generally good: most of the greenhouse area was predicted correctly. There was a false positive greenhouse below left of the detected greenhouses in Niagara S1 (Figure 7). This corresponds to a river-coastal area. Originally this false positive was significantly larger, but increasing the sample size for a coastal urban area (with a fairly straight coastline) significantly reduced the size and also helped with some other areas. If more coastline images were added to the training set (with different river beds, etc.) this error may be further reduced.

For Fraser, the results varied depending on the area. For Fraser N2 (Figure 9) the results were good. The results were not as good for Fraser N3 (Figure 10), as a cluster of small greenhouses right of the detected greenhouses were missed (along with some false positives). For Fraser N5 (Figure 11) a significant number of greenhouses were missed. Various experimentation so far has not improved the results for Fraser. To improve these results, the team would need to investigate what type of greenhouses these are, if additional areas containing these types of greenhouses can be added to the training set, and even if this type of greenhouse can be detected from the 5m satellite images.

Conclusion

Overall, convolutional neural networks were successfully used to detect greenhouses from satellite images in multiple areas. This was particularly true in the areas of Leamington, Niagara, and Fraser. Other areas are still showing low prediction levels for greenhouses. Additionally, there are still issues with small greenhouses in all three areas of interest, which were not large enough to be detected in the 5m RapidEye satellite imagery. These challenges could be solved by higher resolution aerial acquisitions.

The next phase of this project will explore greenhouse detection from higher resolution aerial images. Different methodologies are used when working with higher resolution aerial imagery, for instance the use of UNet-based image segmentation architectures to identify areas corresponding to greenhouses, which we look forward to exploring in a future article.

By: Blair Drummond, Statistics Canada

Statistics Canada is modernizing agriculture data collection by using satellite imagery to predict crop growth. The data scientists were faced with a series of challenges throughout this project, including the seemingly prohibitive expense of scaling the project to production requirements (despite promising initial results). They rose to the occasion by considering all options, including non-obvious ones, and experienced first-hand the value in having a diversely skilled team.

A team of Statistics Canada data scientists, working with experts from the agency's agriculture program, created a successful machine-learning proof-of-concept. They implemented a neural network which achieved 95% accuracy in predicting what crop was growing in a quarter-section (160 acre lot), using freely available satellite imagery. This represented a great opportunity for StatCan's Agriculture program, because the use of satellite imagery offers a way to get mid-season estimates, or even near real-time estimates, and the new approach helps to reduce the response burden for agricultural operators who were required to fill out surveys regularly.

But there was just one problem. The implementation produced by the proof-of-concept had a preprocessing step that extracted pixel data from the Landsat 8 satellite images and applied some transformations. On a single image, this process took approximately one day to complete, used over a 100 gigabytes of RAM, and the cost was estimated at $50 per image on a basic cloud desktop Virtual Machine (VM). The training set consisted of seven years and three provinces worth of data, or around 1,600 images total. If everything worked the first time using available public cloud infrastructure, the project would cost $80,000, before even getting to the step of training the model.

This cost would have been prohibitive for this project, which was still in the experimentation phase. It was not clear if the model would work well on a larger scale than this experiment, and the cost to develop a new model had this substantial hurdle in front of it. The data scientists were fairly confident that they could find a model that would work, but they had to make it more economical. Unphased, they set out on a small proof-of-concept to see if they could get over this hurdle and make the preprocessing steps economically viable.

The proof-of-concept: the first three pipelines

When the data scientists started the pipeline experiment for this project in the fall of 2019, the cloud was still relatively new to them, Data Analytics as a Service (DAaaS) was a young project, and the team was familiarizing themselves with what was newly available. The proof-of-concept was certainly aimed at solving a particular problem, but it was also a division-wide experiment to figure out how to navigate the cloud. For that reason, they took this on as a joint project with the DAaaS team and cloud solution architects. The aim was to get experience with a range of different technologies, including:

• Azure Machine Learning
• Azure Batch
• DAaaS & Kubernetes

The Azure Batch and Azure Machine Learning solutions were to be implemented by a team who had expertise in those, in close collaboration with the Data Science Division (DScD). The DScD worked also with the DAaaS team to see what the DAaaS platform could offer. Each team was handed the same set of code and a couple of test images, and over the span of four months they worked to implement their solutions.

At the end of the implementation period, each approach was analyzed to produce cost-estimates for the processing of a single image (Figure 1).

The pipelines ran on different VM types chosen by the architect. The Azure solutions used low-priority instances, and the Kubernetes solution pricing is based on three-year reserved pricing (which is more expensive than low-priority). The real question became: why such huge differences in cost?

Comparing apples and oranges

The difference was the code. While each team received the same code at the start with the objective of parallelizing it, the Azure Machine Learning and Azure Batch solutions encouraged slightly different approaches to that parallelization. The small changes to the code led to significant differences in outcome. This was not really about the pipeline technology itself; all things being equal the performance would have been comparable, but one implementation bypassed a serious performance issue whereas another did not touch that part of the code.

For example, one of the issues with the original implementation was the way that it parallelized processing of an image. In its original form, it split up what needed to be extracted from the image into 30 groups and then it created 30 parallel processes, each handling a share of the image. On the surface, this was a great idea; but unfortunately there was a complexity. The extraction algorithm needed to load a large geographic data file into memory, as well as the image itself, which combined to about 3 GB of RAM. This would be fine for one process, but since the processes do not share memory, doing this across 30 processes in parallel ballooned the RAM usage to 90 GB. In addition, all processes wrote many small files to disk in the extraction process, and the parallel disk writes slowed down the program substantially. This first implementation used a large number of resources, and took longer than it should have because it was stuck writing data to disk.

This was one area where Azure Machine Learning and Azure Batch diverged. The Azure Batch solution made it convenient to parallelize at the level of those groups within the image to extract, and so those processes were split up across different machines. The RAM was much more manageable, and the processes did not compete while writing to the disk. It was less natural to do this in Azure Machine Learning, and so by no fault of its own, it appeared to have much worse performance.

In contrast, on the DAaaS / Kubernetes implementation, the data scientists carefully read through, rewrote and re-architected components, and, before even touching the pipeline, had an extraction process which:

• Used 6 GB of RAM per image, not 100 GB
• Ran in under 40 minutes, not several hours
• Used 6 CPUs, not 30+

Before even getting to the pipeline, the problem was reduced from one that required large clusters to one that would run on a mid-tier laptop. Since the pipeline no longer needed to constrain its imagination to making one image economical, it was possible to move on to the stage of making the processing of a season's worth of images simple, manageable and automated.

How do we know this now? Why was this not noticed?

The team focused on the Azure Batch and Azure Machine Learning solutions did not have the benefit of a mandate that included redesigning code. They were explicitly not supposed to be modifying the code, as that would have potential methodological impacts, and has little to do with the Cloud solutions or proofs-of-technology. It simply did not fall under their purview.

On the other hand, the DScD team had just hired someone specifically to invest in their data engineering capacity, so in addition to working on a proof-of-technology with DAaaS, the team was also undergoing an effort to develop more expertise in this area. The data engineer undertook a thorough code-review, and had the benefit of having the original author of the code in-house to answer questions. They simply had more freedom to solve the problem in a range of ways by having resources in-house, which led to more efficient outcomes as well as new insights.

It is important to note that without this analysis and review of the code, not only would they not have had this new solution, but they also would not have known why the two Azure solutions had such performance differences! It was only clear after review what caused the differences between the three solutions.

Lessons learned

The team did not compare apples with apples in this situation, and it was a far more informative experience as a result. What they inadvertently compared was really

• what happens when you try to Lift-and-Shift an existing application?

with

• what can you achieve with an analysis on the existing application?

The former is basically a platform/infrastructure question, and the latter is an engineering/application question. What was uncovered in this proof-of-concept was that while the platform is a necessary context, without which this exercise would have been impossible, the real value was delivered by the core engineering^{Footnote 1}, and it was the work invested in the application itself that differentiated the outcomes.

The team was wiser for the experience and learned about where to focus their expertise. Because of this project, and similar experiences, they were able to make strategic decisions in the division about how and where to grow capacity. In the year since the experiment, the DScD's efforts to boost data engineering skills across the division has paid dividends on many projects.

Choosing a pipeline technology and what went wrong the first time

In the previous section it was demonstrated that the underlying pipeline technology was not really what affected the efficiency or cost. So what drove the decisions? And what mistakes were made the first time?

As alluded to earlier, the DScD's business value is in developing models and applications, not in provisioning or maintaining infrastructure. In addition, the division is only a small part of a much larger organization, and it is important that they keep their technology strategy aligned with the organization and leverage the horizontal services offered by solutions like DAaaS.

For the DScD, it was an easy decision. Aligning with and working with DAaaS enables the DScD's work by letting them focus on the things that bring business value to clients, and working with the DAaaS platform helps the DAaaS team build up a robust and flexible platform that meets customer needs—for the DScD, for Statistics Canada, for external partners and ultimately, for Canadians.

Going with the DAaaS / Kubernetes solution was obvious in the end. They implemented a fully automated pipeline which queried the United States Geological Survey Application Programming Interface for the images, downloaded and processed them, and did all of this automatically and in a version-controlled and artifact-driven way. The team had a successful close-out and the team and the clients were very happy with this solution.

Unfortunately, the pipeline ran on a specific piece of pipeline software, and about a month later, that software was moved to a new licensing model that caused a rethink of its use. It was found that continued use of the software was impossible (as purchasing the software was not deemed a viable option). As a result, the rewrite was inevitable anyway.

The rewrite and the benefit of hindsight

In many ways, the rewrite gave the team a chance to revisit and simplify the original implementation. The first pipeline had many small components which each tackled one function, and then the pipeline code orchestrated them.

In the revised pipeline, using Kubeflow Pipelines, the complexity was shifted out of the pipeline orchestration and into the application code itself.

While this sounds counter-intuitive, the reality is that the pipeline is no more or less complicated whether the logic is encoded into the ‘components' of the pipeline, or in the fabric which stitches them together. The difference is more people know regular Python or R than they know pipeline orchestration code, so it is simpler and more maintainable (for projects like this) to not be over-zealous with the pipeline code. As a result, from the Kubeflow Pipelines side, the pipeline looks like this:

It simply gets the list of images that it needs to process that day, fans out in parallel across the Kubernetes cluster, and extracts each image separately. The component itself is encapsulated in a Docker image, which keeps the component portable and makes it easy to test and deploy. The pipeline orchestration code is about 20 lines of Python.

The data flow within the extractor is still a little complicated, but it is easily and efficiently managed within the extractor, using an S3 Bucket (implemented by MinIO) as a storage location and cache.

The team was happy with the result, and successfully processed 1,600 images without issue.

The end?

This is not actually the end of the story as the last chapter has yet to be written. With the new pipeline, the project will be expanded and transitioned to production, and soon the team will begin training the new model.

We look forward to sharing our developments in future articles as the project moves ahead. Who knows, you might get the chance to read about a fancy new neural network that can figure out what is growing from space!