Robotic surveying of fruit plants

Patent metadata

Field	Value
Publication number	US-2016307329-A1
Application number	US-201615131745-A
Country	US
Kind code	A1
Filing date	Apr 18, 2016
Priority date	Apr 16, 2015
Publication date	Oct 20, 2016
Grant date	—

Abstract

A method of machine vision includes identifying contours of fruits in a first image and a second image and performing two-way matching of contours to identify pairs of matched contours, each pair comprising a respective first contour in the first image that matches a respective second contour in the second image. For each pair of matched contours, a respective affine transformation that transforms points in the respective second contour to points in the respective first contour is identified. The second image is mapped to the first image using the affine transformations to form a composite image and the number of fruits in the composite image is counted.

First claim

What is claimed is:

1 . A method of machine vision, the method comprising:
identifying contours of fruits in a first image and a second image; performing two-way matching of contours to identify pairs of matched contours, each pair comprising a respective first contour in the first image that matches a respective second contour in the second image; for each pair of matched contours, identifying a respective affine transformation that transforms points in the respective second contour to points in the respective first contour; mapping the second image to the first image using the affine transformations to form a composite image; and counting fruits in the composite image.

2 . The method of claim 1 wherein mapping the second image to the first image using the affine transformations comprises determining a homography between the second image and the first image based in part on the affine transformations and using the homography to map the second image onto the first image.

3 . The method of claim 1 wherein identifying contours of fruits in the first and second images comprise filtering the first and second images to remove at least one pixel value based on the colors represented by the pixel value.

4 . The method of claim 1 wherein performing two-way matching comprises selecting a test contour in the first image, searching the second image to identify a second image matching contour that matches the test contour, searching the first image to identify a first image matching contour that matches the second image matching contour, and designating the test contour and second image matching contour as a pair of matched contours when the first image matching contour is the test contour.

5 . The method of claim 1 further comprising mapping multiple images to the composite image using the affine transformations.

6 . The method of claim 5 further comprising forming multiple composite images and counting fruits in each composite image.

7 . The method of claim 6 wherein forming multiple composite images comprises comparing contours in a third image to contours in the first image and when none of the contours of the third image are in the first image, designating the third image as a composite image.

8 . A method comprising:
receiving a plurality of images; forming a plurality of composite images from the plurality of images, each composite image formed by mapping pixels from multiple images onto the composite image; counting fruit in each composite image to form a fruit count for each composite image; and summing the fruit counts to form a total fruit count.

9 . The method of claim 8 wherein forming a plurality of composite images comprises:
setting a first image as a composite image;
comparing a successive image to the first image and:
mapping pixels of the successive images to the first image when at least one contour in the successive image matches a contour in the first image; and
setting the successive image as a new composite image when none of the contours of the successive image matches a contour in the first image.

10 . The method of claim 9 wherein a plurality of images are mapped to the first image to form the composite image.

11 . The method of claim 9 wherein comparing the successive image to the first image comprises:
identifying contours in the first image;
identifying contours in the successive image;
selecting a contour in the first image;
searching for and finding a matching contour in the successive image that matches the selected contour;
searching for and finding a first image matching contour in the first image that matches the matching contour of the successive image; and
determining that the first image matching contour is the selected contour.

12 . The method of claim 11 wherein identifying contours in the first image and the successive image comprises filtering pixels based on color values for the pixels to locate pixels that are likely to represent fruit and identifying contours within the located pixels.

13 . The method of claim 8 wherein counting fruit in a composite image comprises applying a Gaussian mixture model to the composite image.

14 . The method of claim 8 wherein forming a composite image comprises determining a separate affine transformation for each of a plurality of contours in one of the plurality of images.

15 . A system comprising:
an unmanned aerial vehicle having at least one camera that captures multiple images of an agricultural plot; a server that receives the captured multiple images and processes the captured multiple images to count fruit in the agricultural plot, wherein the server processes the captured multiple images by:
forming a plurality of composite images, each composite image formed by mapping a respective plurality of the captured multiple images to the composite image;
counting fruit in the composite images to form a separate count for each composite image; and
summing the separate counts to form a total fruit count.

16 . The system of claim 15 forming a plurality of composite images comprises comparing a first image to a second images and creating a new composite image from the second image when none of the contours of the second image have a corresponding contour in the first image.

17 . The system of claim 16 wherein determining if a contour in the second image has a corresponding contour in the first image comprises searching for a corresponding contour in the first image and after finding a corresponding contour in the first image, searching the second image for a contour corresponding to the corresponding contour of the first image.

18 . The system of claim 15 wherein counting fruit in a composite image comprises applying a Gaussian Mixture Model to the composite image.

19 . The system of claim 15 wherein mapping a captured image onto the composite image comprises determining a respective affine transformation for each of a plurality of contours in the composite image, using the affine transformation to identify points of correspondence between the captured image and the composite image and mapping the captured image onto the composite image based on the points of correspondence.

20 . The system of claim 19 wherein mapping the captured image onto the composite image comprises setting pixels in the captured image and the composite image to zero if the pixels are unlikely to represent fruit and not mapping pixels in the captured image to the composite image if there is a non-zero pixel in the composite image where the pixel in the captured image would be mapped to.

Assignees

• Univ Minnesota

Inventors

• Isler Volkan
• Tokekar Pratap Rajkumar
• Stefas Nikolaos
• Roy Pravakar

Classifications

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06V10/752

  Contour matching · CPC title

• G06T7/0085 Primary

  Physics · mapped topic

• G06K9/3216

  Physics · mapped topic

• G06K9/3208

  Physics · mapped topic