Semantic segmentation to identify and treat plants in a field and verify the plant treatments

Patent metadata

Field	Value
Publication number	US-11514671-B2
Application number	US-202016893405-A
Country	US
Kind code	B2
Filing date	Jun 4, 2020
Priority date	May 24, 2018
Publication date	Nov 29, 2022
Grant date	Nov 29, 2022

Abstract

A farming machine including a number of treatment mechanisms treats plants according to a treatment plan as the farming machine moves through the field. The control system of the farming machine executes a plant identification model configured to identify plants in the field for treatment. The control system generates a treatment map identifying which treatment mechanisms to actuate to treat the plants in the field. To generate a treatment map, the farming machine captures an image of plants, processes the image to identify plants, and generates a treatment map. The plant identification model can be a convolutional neural network having an input layer, an identification layer, and an output layer. The input layer has the dimensionality of the image, the identification layer has a greatly reduced dimensionality, and the output layer has the dimensionality of the treatment mechanisms.

First claim

What is claimed is:

1. A method for identifying a plant in a field, the method comprising:
accessing an image of the field captured by an imaging system, the accessed image having a pixel dimensionality and including a plurality of pixels with a subset of the pixels representing the plant in the field;
accessing a semantic segmentation model trained to identify plants in images by recognizing pixels in images representing plants, the semantic segmentation model trained with previously captured images comprising green plants and labelled to identify green pixels representing the green plants;
applying the semantic segmentation model to the image to identify which pixels of the image represent the plant, the semantic segmentation model configured to:
identify the subset of pixels in the image representing the plant,
generate a treatment map comprising a plurality of elements, each element of the plurality corresponding to a group of pixels of the plurality of pixels, one or more of the elements corresponding to the subset of pixels representing the plant, and each element indicating a treatment for a portion of the field represented by pixels corresponding to that element, and
wherein a dimensionality of the plurality of elements in the treatment map is less or equal to the pixel dimensionality.

2. The method of claim 1 , further comprising:
storing the treatment map in a treatment map database accessible by a farming machine.

3. The method of claim 1 , further comprising:
treating the plant in the field according to the treatment map.

4. The method of claim 1 , wherein the method is performed by a farming machine, and the farming machine comprises a plurality of treatment mechanisms for treating plants, the plurality of treatment mechanisms having a treatment mechanism dimensionality that is the same as the treatment map dimensionality.

5. The method of claim 1 , further comprising:
for each element in the plant treatment map, generating a plant treatment instruction corresponding to the treatment indicated by the element comprising the subset of pixels identified as representing the plant;
storing the plant treatment instructions in an instruction database.

6. The method of claim 5 , further comprising:
transmitting the plant treatment instructions to the farming machine, the farming machine comprising a plurality of plant treatment mechanisms configured to actuate the plant treatment mechanisms using the plant treatment instruction to treat the identified plant based on the treatment map.

7. The method of claim 1 , wherein the semantic segmentation model further comprises:
a plurality of neural network layers comprising:
an input layer comprising a plurality of input nodes, a number of the plurality of input nodes corresponding to the pixel dimensionality,
an output layer comprising a plurality of output nodes, a number of the plurality of output nodes corresponding to the treatment map dimensionality, and
one or more intermediate layers comprising a plurality of intermediate nodes, the nodes of the intermediate layers transforming data in the input layer to data in the output layer.

8. The method of claim 1 , wherein applying the semantic segmentation model further comprises:
accessing a treatment array including a plurality of treatment elements, wherein:
the treatment array has the treatment dimensionality,
each of the plurality of treatment elements corresponds to an element of the treatment map, and
each of the plurality of treatment elements corresponds to a treatment area of a plant treatment mechanism of a farming machine.

9. The method of claim 8 , further comprising:
mapping the treatment array to the accessed image such that each element of the treatment array corresponds to a group of pixels in the accessed image representing the treatment area of the corresponding treatment mechanism.

10. The method of claim 9 , further comprising:
mapping the identified plants to elements of the treatment array to generate a treatment map such that each element of the treatment map includes the identified plant when the identified plant is in its corresponding treatment area.

11. The method of claim 1 , wherein:
the treatment dimensionality is a representation of any of the number, size, shape, arrangement, and configuration of the treatments applied by a plurality of plant treatment mechanisms.

12. The method of claim 1 , wherein the farming machine is any of a crop sprayer, a tractor, a tiller, or a seeder.

13. The method of claim 1 , wherein the treatment is a spray treatment from an array of spray nozzles.

14. The method of claim 13 , wherein the spray treatment is water, nutrients, a herbicide, a fungicide, or another pesticide.

15. A non-transitory computer readable storage medium comprising instructions for identifying a plant in a field, the instructions, when executed by one or more processors, causing the one or more processors to:
access an image of the field captured by an imaging system, the accessed image having a pixel dimensionality and including a plurality of pixels with a subset of the pixels representing the plant in the field;
access a semantic segmentation model trained to identify plants in images by recognizing pixels in images representing plants, the semantic segmentation model trained with previously captured images comprising green plants and labelled to identify green pixels representing the green plants;
apply a semantic segmentation model to the image to identify which pixels of the image represent the plant, the semantic segmentation model configured to:
identify the subset of pixels in the image representing the plant,
generate a treatment map comprising a plurality of elements, each element of the plurality corresponding to a group of pixels of the plurality of pixels, one or more of the elements corresponding to the subset of pixels representing the plant, and each element indicating a treatment for a portion of the field represented by pixels corresponding to that element, and
wherein a dimensionality of the plurality of elements in the treatment map is less or equal to the pixel dimensionality.

16. The non-transitory computer readable storage medium of claim 15 , wherein the instructions, when executed, cause the one or more processors to:
store the treatment map in a treatment map database accessible by a farming machine.

17. The non-transitory computer readable storage medium of claim 15 , wherein the instructions, when executed, cause the one or more processors to:
treat the plant in the field according to the treatment map, the treatment map executed by a farming machine comprising a plurality of treatment mechanisms configured to receive and execute the treatment map.

18. A farming machine comprising:
a non-transitory computer readable storage medium comprising instructions for identifying a plant in a field, the instructions, when executed by one or more processors, causing the one or more processors to:
access an image of the field captured by an imaging system, the accessed image having a pixel dimensionality and including a plurality of pixels with a subset of the pixels representing the plant in the field;
access a semantic segmentation model trained to identify plants in images by recognizing pixels in images representing plants, the semantic segmentation model trained with previously captured images comprising green plants and labelled to identify green pixels representing the green plants;
apply a semantic segmentation model to the image to identify which pi

Assignees

• Blue River Tech Inc

Inventors

• Polzounov Andrei
• Ostrowski James Patrick
• Redden Lee Kamp
• Denas Olgert
• Fu Chia-Chun
• Padwick Chris

Classifications

• G06V20/38

  Outdoor scenes · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06V20/188 Primary

  Vegetation · CPC title

• G06N3/045

  Combinations of networks · CPC title