Fruit perception system for robotic harvesting

Patent metadata

Field	Value
Publication number	US-2025081892-A1
Application number	US-202318463516-A
Country	US
Kind code	A1
Filing date	Sep 8, 2023
Priority date	Sep 8, 2023
Publication date	Mar 13, 2025
Grant date	—

Abstract

The fruit perception system for a robotic harvester acquires and processes fruit detection and localization information to enable a robotic manipulator arm to connect with a targeted fruit. The system includes multiple embodiments, but comprises at least one RGB-D camera and one horizontally slidable line scan laser. The system uses the RGB-D camera data and a planning algorithm to identify a specific target fruit to pick first. The line scan laser paints the surface of the target fruit with a laser line and the RGB-D camera extracts line scan laser image data and communicates the line scan laser image data to a controller/processor. The controller/processor processes the extracted laser line image data and determines the xyz position of the centroid of the target fruit, so that the controller/processor directs a manipulator arm to pick the fruit.

First claim

What is claimed is:

1 . A robotic harvester having a fruit perception system comprising:
a slide mechanism; a line scan laser that is selectively movable by the slide mechanism; an RGB-D camera, the line scan laser projecting a laser line within a field of view of the RGB-D camera; a controller/processor in communication with the slide mechanism, the line scan laser, and the RGB-D camera, the controller/processor directing operations and movements of the line scan laser, and processing images from the RGB-D camera; wherein as the slide mechanism moves the line scan laser, the line scan laser projects a laser line onto a target fruit from multiple sequential positions, the RGB-D camera acquires images of the laser line projected onto the target fruit, and communicates the laser line images to the controller/processor, the controller/processor processing the laser line images and determining an xyz position of the centroid of the target fruit.

2 . The perception system of claim 1 wherein the perception system is in electrical communication with a manipulator arm through the controller/processor, the controller/processor directing the manipulator arm to the target fruit.

3 . The perception system of claim 2 wherein the manipulator arm further comprises an end effector which makes at least a partial vacuum seal.

4 . The perception system of claim 2 wherein the perception system is positioned above the manipulator arm.

5 . The perception system of claim 3 wherein the RGB-D camera is positioned and structured to acquire detection and localization data to direct the manipulator arm and associated end effector to the centroid of the target fruit and thereby pick the target fruit.

6 . The perception system of claim 1 wherein the slide mechanism selectively moves the line scan laser horizontally by means of an electrically controllable motor that is in communication with the controller/processor, while the RGB-D camera remains stationary.

7 . The perception system of claim 1 wherein the perception system is structured so that the RGB-D camera is positioned adjacent to the line scan laser and the slide mechanism.

8 . The perception system of claim 1 further comprising an RGB camera, the perception system being structured so that the RGB-D camera acquires fruit detection data, while the RGB camera acquires fruit localization data, both the RGB and the RGB-D cameras being in communication with the controller/processor.

9 . The perception system of claim 1 wherein the perception system comprises an additional line scan laser so that the perception system comprises two separate line scan lasers, each of the line scan lasers being connected to a corresponding separate slide mechanism so that each line scan laser moves independently from the other line scan laser.

10 . The perception system of claim 9 wherein the system is structured so that the RGB-D camera communicates image data from both line scan lasers to the controller/processor-which communicates the data to two separate manipulator arms.

11 . The perception system of claim 8 wherein the perception system is structured so that the RGB-D camera is positioned between the two line scan lasers.

12 . A method of robotically picking a target fruit, the method comprising:
(a) providing the perception system of claim 1 ; (b) directing the RGB-D camera to detect candidate target fruits and acquire rough images identifying the candidate target fruits and defining the candidate target fruits by bounding boxes; (c) using a planning algorithm to select the target fruit from the candidate fruits in the bounding boxes; (d) actuating the slide mechanism to direct the line scan laser to an initial position to paint the target fruit with a laser line; (e) utilizing the RGB-D camera to obtain an RGB image of the line scan laser line as it paints a surface of the target fruit, and communicating the line scan laser image data to the controller/processor; (f) utilizing image processing algorithms to extract the laser line on the target fruit from the red-channel of the RGB image; (g) utilizing a laser triangulation technique to determine the xyz position of the extracted laser line; (h) moving the line scan laser to a next position and painting the target fruit surface with a laser line, the RGB-D camera communicating extracted line scan laser image data to the controller/processor; (i) repeating steps (f) and (g) until sufficient line scan laser image data is acquired for the controller/processor to calculate the xyz positions of all laser lines; (j) determining the xyz position of the centroid of the target fruit by selecting one of the laser lines based on a holistic evaluation function.

13 . The method of claim 12 wherein, in step (b), a deep learning algorithm is used to detect all fruits in the workspace and define the bounding boxes.

14 . The method of claim 12 wherein, in step (c), image depth point cloud data and bounding boxes are used in combination with the planning algorithm to identify the target fruit.

15 . The method of claim 12 wherein, in step (d), the line scan laser paints a left half region of the target fruit with a red laser line.

16 . The method of claim 12 wherein, in step (h), the line scan laser moves horizontally.

17 . The method of claim 16 wherein the line scan laser moves to a first position, and then moves to about four sequential positions, the RGB-D camera extracting line scan laser image data at each position.

18 . The method of claim 17 wherein the line scan laser moves in about 1 cm increments.

19 . The method of claim 12 wherein, in step (a), the perception system includes an additional RGB camera.

20 . The method of claim 19 wherein, in steps (c) and (h) utilizing the RGB camera (rather than the RGB-D camera) extracts line scan laser image data from the line scan laser line as the laser line paints the surface of the target fruit.

Assignees

• Us Agriculture
• Univ Michigan State

Inventors

• Lu Renfu
• Li Zhaojian
• Zhang Kaixiang
• Lammers Kyle
• Chu Pengyu

Classifications

• B25J9/1697

  Vision controlled systems · CPC title

• A01D46/30 Primary

  Robotic devices for individually picking crops · CPC title

• A01D46/24 Primary

  Devices for picking apples or like fruit (A01D46/28 {and A01D46/005} takes precedence) · CPC title