Automated plant detection using image data

Patent metadata

Field	Value
Publication number	US-11748976-B2
Application number	US-202117378658-A
Country	US
Kind code	B2
Filing date	Jul 17, 2021
Priority date	May 9, 2017
Publication date	Sep 5, 2023
Grant date	Sep 5, 2023

Abstract

A plant treatment platform uses a plant detection model to detect plants as the plant treatment platform travels through a field. The plant treatment platform receives image data from a camera that captures images of plants (e.g., crops or weeds) growing in the field. The plant treatment platform applies pre-processing functions to the image data to prepare the image data for processing by the plant detection model. For example, the plant treatment platform may reformat the image data, adjust the resolution or aspect ratio, or crop the image data. The plant treatment platform applies the plant detection model to the pre-processed image data to generate bounding boxes for the plants. The plant treatment platform then can apply treatment to the plants based on the output of the machine-learned model.

First claim

What is claimed is:

1. A method comprising:
capturing a first image from a camera mounted on a plant treatment platform passing through or over a field, the captured first image comprising image data representing an area of the field;
inputting, with a computer, a second image based on the captured first image into a plant detection model to:
generate a plant bounding box within the second image that encloses image data representing an individual plant growing in the area of the field and a region of the field surrounding the individual plant,
determine a location of the plant bounding box in the second image, and
identify a physical location of the individual plant based on the location of the plant bounding box in the second image; and

treating, with a treatment mechanism, at least the individual plant enclosed by the plant bounding box by dispensing a treatment at the identified physical location.

2. The method of claim 1 , further comprising: applying, with the computer, a pre-processing function to the captured first image to generate the second image for processing by the plant detection model.

3. The method of claim 2 , wherein the pre-processing function prepares the image data for processing by the plant detection model.

4. The method of claim 2 , wherein applying the pre-processing function includes debayering the image data of the captured first image.

5. The method of claim 2 , wherein applying the pre-processing function includes cropping the image data of the captured first image.

6. The method of claim 2 , wherein applying the pre-processing function includes white balancing image data of the captured first image, wherein the white balancing is based on at least one of: a time of day the camera captured the captured first image, whether the camera used artificial lighting to capture the captured first image, or whether the camera used a shroud to block or diffuse sunlight.

7. The method of claim 2 , wherein applying the pre-processing function includes resizing the captured first image.

8. The method of claim 2 , wherein applying the pre-processing function includes adjusting an exposure of image data of the captured first image.

9. The method of claim 2 , wherein applying the pre-processing function includes normalizing values of the image data of the captured first image.

10. The method of claim 1 , wherein the plant detection model comprises one or more submodels, each submodel of the one or more submodels identifying a different plant species, and wherein the plant bounding box is generated using a submodel of the one or more submodels and comprises an identifier for a plant species boxed by the plant bounding box.

11. The method of claim 1 , wherein the plant bounding box comprises a measure of confidence representing a likelihood that the plant bounding box boxes a plant in the field.

12. The method of claim 1 , wherein the plant detection model comprises a modified version of a Single Shot MultiBox Detector model.

13. The method of claim 12 , wherein the plant detection model uses at least one of the following techniques: batch normalization, leaky rectified linear units, residual neural networks, custom anchor boxes, cleaned labeled data, increased spatial resolution on feature maps, spatial transformers, training loss optimization, or weighted softmax.

14. The method of claim 1 , further comprising:
transmitting instructions to the treatment mechanism to treat the individual plant based on the generated plant bounding box, the instructions causing the treatment mechanism to:
position the treatment mechanism in a direction of the individual plant;
select a treatment fluid based on a species of the individual plant; and
dispense the treatment fluid onto the individual plant boxed by the plant bounding box.

15. The method of claim 1 , wherein the plant detection model is trained based on labeled image data, wherein the labeled image data includes images with plant bounding boxes.

16. A non-transitory computer-readable storage medium comprising stored instructions that, when executed by a computer, cause the computer to perform operations including:
capturing a first image from a camera mounted on a plant treatment platform passing through or over a field, the captured first image comprising image data representing an area of the field;
inputting a second image based on the captured first image into a plant detection model to:
generate a plant bounding box within the second image that encloses image data representing an individual plant growing in the area of the field and a region of the field surrounding the individual plant,
determine a location of the plant bounding box in the second image, and
identify a physical location of the individual plant based on the location of the plant bounding box in the second image; and

treating, with a treatment mechanism, at least the individual plant enclosed by the plant bounding box by dispensing a treatment at the identified physical location.

17. The non-transitory computer-readable storage medium of claim 16 , further comprising: applying, with the computer, a pre-processing function to the captured first image to generate the second image for processing by the plant detection model.

18. The non-transitory computer-readable storage medium of claim 17 , wherein the pre-processing function prepares the image data for processing by the plant detection model.

19. The non-transitory computer-readable storage medium of claim 16 , wherein the plant bounding box comprises a measure of confidence representing a likelihood that the plant bounding box boxes a plant in the field.

20. A farming machine comprising:
an imaging sensor mounted to the farming machine, the imaging sensor configured to capture a first image as the farming machine passes through or over a field, the captured first image comprising image data representing an area of the field;
a computer; and
a non-transitory computer readable medium comprising instructions that, when executed by the computer, cause the computer to perform operations including:
inputting a second image based on the captured first image into a plant detection model to:
generate a plant bounding box within the second image that encloses image data representing an individual plant growing in the area of the field and a region of the field surrounding the individual plant,
determine a location of the plant bounding box in the pre-processed second image, and
identify a physical location of the individual plant based on the location of the plant bounding box in the second image; and

treating, with a treatment mechanism, at least the individual plant enclosed by the plant bounding box by dispensing a treatment at the identified physical location.

Assignees

• Blue River Tech Inc

Inventors

• Redden Lee Kamp
• Padwick Christopher Grant
• Radhakrishnan Rajesh
• Ostrowski James Patrick

Classifications

• G06V10/764 Primary

  using classification, e.g. of video objects · CPC title

• A01M7/0089

  Regulating or controlling systems (the delivery being related to the movement of a vehicle B05B9/06) · CPC title

• G06F18/2414

  Smoothing the distance, e.g. radial basis function networks [RBFN] · CPC title

• G06T7/0002

  Inspection of images, e.g. flaw detection · CPC title

• G06T7/75

  involving models · CPC title