Semantic segmentation to identify and treat plants in a field and verify the plant treatments

Patent metadata

Field	Value
Publication number	US-10713484-B2
Application number	US-201816126842-A
Country	US
Kind code	B2
Filing date	Sep 10, 2018
Priority date	May 24, 2018
Publication date	Jul 14, 2020
Grant date	Jul 14, 2020

Abstract

A farming machine including a number of treatment mechanisms treats plants according to a treatment plan as the farming machine moves through the field. The control system of the farming machine executes a plant identification model configured to identify plants in the field for treatment. The control system generates a treatment map identifying which treatment mechanisms to actuate to treat the plants in the field. To generate a treatment map, the farming machine captures an image of plants, processes the image to identify plants, and generates a treatment map. The plant identification model can be a convolutional neural network having an input layer, an identification layer, and an output layer. The input layer has the dimensionality of the image, the identification layer has a greatly reduced dimensionality, and the output layer has the dimensionality of the treatment mechanisms.

First claim

What is claimed is:

1. A method for treating a plant in a field as a farming machine travels through the field, the farming machine including an imaging system to capture images of plants in the field as the farming machine travels through the field, the method comprising:
accessing an image captured by the imaging system, the accessed image having a pixel dimensionality and including pixels representing the plant in the field;
applying a semantic segmentation model to the image to identify which pixels of the image represent the plant, the semantic segmentation model including a plurality layers in a convolutional neural network, and the semantic segmentation model:
encoding the image onto a first layer as an encoded image, the first layer having the pixel dimensionality;
reducing the encoded image to a reduced image, the reduced image on a second layer and having an identification dimensionality;
identifying latent features in the reduced image as the plant;
decoding the reduced image to a treatment map, the treatment map on a third layer having a dimensionality substantially similar to a dimensionality of a plurality of plant treatment mechanisms coupled to the farming machine, and
identifying elements of the treatment map including the plant, each element corresponding to a plant treatment mechanism;

generating a set of plant treatment instructions for the plurality of plant treatment mechanisms based on identified elements of the treatment map including the plant;
sending the plant treatment instructions to the plurality of plant treatment mechanisms; and
actuating the plant treatment mechanisms using the plant treatment instructions such that the identified plant is treated with the plant treatment mechanisms.

2. The method of claim 1 , wherein:
the dimensionality of the plurality of plant treatment mechanisms is a representation of any of a number, a size, a shape, an arrangement, or a configuration of the plurality of plant treatment mechanisms.

3. The method of claim 1 , wherein the pixel dimensionality is a number of pixels in the accessed image.

4. The method of claim 1 , wherein the pixel dimensionality is greater than the identification dimensionality.

5. The method of claim 4 , wherein the treatment dimensionality is greater than the identification dimensionality and less than the pixel dimensionality.

6. The method of claim 1 , wherein the farming machine is any of a crop sprayer, a tractor, a tiller, or a seeder.

7. The method of claim 1 , wherein the plurality of treatment mechanisms comprises an array of spray nozzles.

8. The method of claim 1 , wherein treating the plant can include spraying the plant with any of an herbicide, a fungicide, water, or a pesticide.

9. The method of claim 1 , wherein the semantic segmentation model can identify any of a crop, a weed, soil, a result of a plant treatment, or a condition of a plant.

10. The method of claim 1 , wherein applying the semantic segmentation model further comprises:
accessing a treatment array including a plurality of treatment elements, wherein:
the treatment array has the treatment dimensionality,
each of the plurality of treatment elements corresponds to a plant treatment mechanism of the plurality of plant treatment mechanisms
each of the plurality of treatment elements corresponds to a treatment area of its corresponding plant treatment mechanism.

11. The method of claim 10 , wherein applying the semantic segmentation further comprises:
mapping the treatment array to the accessed image such that each element of the treatment array corresponds to a group of pixels in the accessed image representing the treatment area of the corresponding treatment mechanism.

12. The method of claim 10 , wherein decoding the reduced image to a treatment map further comprises:
mapping the identified plants to elements of treatment array to generate a treatment map such that each element of the treatment map includes the identified plant when the identified plant is in its corresponding treatment area.

13. The method of claim 1 , wherein reducing the encoded image to a reduced image further comprises:
applying, to the encoded image, a transformation function including a set of weights and parameters when reducing the encoded image.

14. The method of claim 1 , wherein decoding the treatment map from the reduced image further comprises:
applying, to the reduced image, a transformation function including a set of weights and parameters when reducing the encoded image.

15. The method of claim 1 , wherein the semantic segmentation includes a number of hidden layers of the plurality of layers, the plurality of hidden layers for executing functions to identify latent information representing the plant and to map the reduced image to the treatment map.

16. A non-transitory computer readable storage medium comprising instructions for treating a plant in a field as a farming machine travels through the field, the farming machine including an imaging system to capture images of plants in the field as the farming machine travels through the field, the instructions, when executed by a processor, causing the processor to:
access an image captured by the imaging system, the accessed image having a pixel dimensionality and including pixels representing the plant in the field;
apply a semantic segmentation model to the image to identify which pixels of the image represent the plant, the semantic segmentation model including a plurality layers in a convolutional neural network, and the semantic segmentation model:
encoding the image onto a first layer as an encoded image, the first layer having the pixel dimensionality;
reducing the encoded image to a reduced image, the reduced image on a second layer and having an identification dimensionality;
identifying latent features in the reduced image as the plant;
decoding the reduced image to a treatment map, the treatment map on a third layer having a dimensionality substantially similar to a dimensionality of a plurality of plant treatment mechanisms coupled to the farming machine, and
identifying elements of the treatment map including the plant, each element corresponding to a plant treatment mechanism;

generate a set of plant treatment instructions for the plurality of plant treatment mechanisms based on identified elements of the treatment map including the plant;
send the plant treatment instructions to the plurality of plant treatment mechanisms; and
actuate the plant treatment mechanisms using the plant treatment instructions such that the identified plant is treated with the plant treatment mechanisms.

17. A farming machine comprising:
a plurality of plant treatment mechanisms for treating a plant as the farming machine travels past the plant in a field;
an image acquisition system to capture an image of the plant in the field as the farming machine moves through the field, the image having a pixel dimensionality and comprising pixels representing the plant in the field; and
a control system including for applying, using a processor, a semantic segmentation model to the image to identify which pixels of the image represent the plant, the semantic segmentation model including a plurality layers in a convolutional neural network, and the semantic segmentation model:
encoding the image onto a first layer as an encoded image, the first layer having the pixel dimensionality;
reducing the encoded image to a reduced image, the reduced image on a second layer and having an identification dimensionality;
identifying latent features in the reduce

Assignees

• Blue River Tech Inc

Inventors

• Polzounov Andrei
• Ostrowski James Patrick
• Redden Lee Kamp
• Denas Olgert
• Fu Chia-Chun
• Padwick Chris

Classifications

• G06V20/38

  Outdoor scenes · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V20/188

  Vegetation · CPC title

• G06N3/08 Primary

  Learning methods · CPC title