Semantic segmentation to identify and treat plants in a field and verify the plant treatments

Patent metadata

Field	Value
Publication number	US-12367389-B2
Application number	US-202217985766-A
Country	US
Kind code	B2
Filing date	Nov 11, 2022
Priority date	May 24, 2018
Publication date	Jul 22, 2025
Grant date	Jul 22, 2025

Abstract

A farming machine including a number of treatment mechanisms treats plants according to a treatment plan as the farming machine moves through the field. The control system of the farming machine executes a plant identification model configured to identify plants in the field for treatment. The control system generates a treatment map identifying which treatment mechanisms to actuate to treat the plants in the field. To generate a treatment map, the farming machine captures an image of plants, processes the image to identify plants, and generates a treatment map. The plant identification model can be a convolutional neural network having an input layer, an identification layer, and an output layer. The input layer has the dimensionality of the image, the identification layer has a greatly reduced dimensionality, and the output layer has the dimensionality of the treatment mechanisms.

First claim

What is claimed is:

1. A method for identifying an implemented treatment for a plant in a field, the method comprising:
accessing an image of the field captured by an imaging system coupled to a farming machine, the image accessed after the farming machine executes the implemented treatment for the plant in the field, and the image comprising a plurality of pixels, the plurality of pixels comprising a first subset of pixels representing the plant in the field and a second subset of pixels representing the implemented treatment in the field;
accessing a semantic segmentation model trained to identify implemented treatments in images by recognizing pixels in images representing the implemented treatment;
applying the semantic segmentation model to the image to identify the implemented treatment by determining the second subset of pixels in the image represents the implemented treatment;
identifying a dissimilarity between the implemented treatment and an expected treatment for the plant; and
identifying a treatment mechanism causing the dissimilarity based on differences between the implemented treatment and the expected treatment for the plant.

2. The method of claim 1 , wherein the implemented treatment is a spray treatment sprayed onto the plant by the farming machine.

3. The method of claim 1 , wherein the expected treatment is a spray treatment expected to be sprayed onto the plant by the farming machine.

4. The method of claim 1 , further comprising:
accessing a treatment map configured to cause the farming machine to execute treatments for plants in the field, where:
the treatment map is generated by applying a second semantic segmentation model to images of the field,
the second segmentation model is trained to identify plants in the field and generate treatment areas for the treatment map, and
each of the treatment areas in the treatment map corresponds to a treatment mechanism of the farming machine; and

applying treatments to plants in the field according to the accessed treatment map.

5. The method of claim 4 , wherein identifying a dissimilarity between the identified treatment and the expected treatment further comprises:
applying the second segmentation model to the image to generate an implemented treatment map, the implemented treatment map identifying treatment areas in the field including implemented treatments, and each of the treatment areas in the implemented treatment map corresponding to a different treatment mechanism of the farming machine; and
comparing the implemented treatment map to the executed treatment map to determine the dissimilarity and identify the treatment mechanism causing the dissimilarity.

6. The method of claim 1 , further comprising:
accessing a second image of the field captured by the imaging system coupled to the farming machine after the farming machine, the accessed second image comprising a plurality of pixels which includes a third subset of pixels representing the plant in the field; and
applying a second semantic segmentation model to identify the plant in the field by determining the third subset of pixels in the second image represents the plant.

7. The method of claim 1 , further comprising:
in response to identifying the treatment mechanism causing the dissimilarity, re-calibrating a control system of the farming machine to correct for the determined dissimilarity.

8. The method of claim 1 , further comprising:
in response to identifying the treatment mechanism causing the dissimilarity, transmitting a notification to an operator of the farming machine indicating the identified treatment mechanism causing the dissimilarity.

9. The method of claim 1 , wherein identifying the dissimilarity further comprises:
identifying a treatment time delay for the treatment mechanism based on differences between the expected treatment and the implemented treatment.

10. The method of claim 1 , wherein identifying the dissimilarity further comprises:
identifying a reduced flow rate for the treatment mechanism based on differences between the expected treatment and the implemented treatment.

11. The method of claim 1 , wherein identifying the dissimilarity further comprises:
identifying a misalignment for the treatment mechanism based on differences between the expected treatment and the implemented treatment.

12. The method of claim 1 , wherein identifying the dissimilarity further comprises:
identifying a clog for the treatment mechanism based on differences between the expected treatment and the implemented treatment.

13. The method of claim 1 , wherein identifying the dissimilarity further comprises:
identifying an incorrect direction of motion for the farming machine based on differences between the expected treatment and the implemented treatment.

14. The method of claim 1 , wherein determining the dissimilarity and identifying the treatment mechanism occurs on a same pass through the field as when the farming machine implements the implemented treatment.

15. The method of claim 1 , wherein determining the dissimilarity and identifying the treatment mechanism occurs on a different pass through the field as when the farming machine implements the implemented treatment.

16. A farming machine comprising:
an imaging system coupled to the farming machine configured to capture images of a field after the farming machine executes an implemented treatment for a plant in the field;
a plurality of treatment mechanisms configured for executing plant treatments in the field;
one or more processors; and
a non-transitory computer readable storage medium storing instructions for identifying the implemented treatment, the instructions, when executed by the one or more processors, causing the one or more processors to:
access an image of the field captured by the imaging system, the image comprising a plurality of pixels and the plurality of pixels comprising a first subset of pixels representing the plant in the field and a second subset of pixels representing the implemented treatment in the field;
access a semantic segmentation model trained to identify implemented treatments in images by recognizing pixels in images representing the implemented treatment;
apply the semantic segmentation model to the image to identify the implemented treatment by determining the second subset of pixels in the image represents the implemented treatment;
identify a dissimilarity between the implemented treatment and an expected treatment for the plant; and
identify a treatment mechanism causing the dissimilarity based on differences between the implemented treatment and the expected treatment for the plant.

17. The farming machine of claim 16 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to:
access a treatment map configured to cause the farming machine to execute treatments for plants in the field, where:
the treatment map is generated by applying a second semantic segmentation model to images of the field,
the second segmentation model is trained to identify plants in the field and generate treatment areas for the treatment map, and
each of the treatment areas in the treatment map corresponds to a treatment mechanism of the farming machine; and

apply treatments to plants in the field according to the accessed treatment map.

18. The farming machine of claim 16 , wherein identifying a dissimilarity between the identified treatment and the expected treatment further causes the one or more processors to:
apply the second segm

Assignees

• Deere & Co

Inventors

• Polzounov Andrei
• Ostrowski James Patrick
• Redden Lee Kamp
• Denas Olgert
• Fu Chia-Chun
• Padwick Chris

Classifications

• G06N3/08 Primary

  Learning methods · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06V10/26

  Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion · CPC title