Method and device for collaborative servo control of motion vision of robot in uncalibrated agricultural scene

Patent metadata

Field	Value
Publication number	US-12162170-B2
Application number	US-201917602903-A
Country	US
Kind code	B2
Filing date	Nov 18, 2019
Priority date	Apr 11, 2019
Publication date	Dec 10, 2024
Grant date	Dec 10, 2024

Abstract

A device and method for collaborative servo control of motion vision of a robot in an uncalibrated agricultural scene is provided. The device includes a robot arm, a to-be-gripped target object, an image sensor and a control module. An end of a robot arm is provided with a mechanical gripper, and a to-be-gripped target object is within a grip range of the robot arm. A control module drives the mechanical gripper to grip the to-be-gripped target object, and controls an image sensor to perform image sampling on a process of gripping the to-be-gripped target object by the robot arm. The image sensor sends sampled image data to the control module. The device does not need to perform precise spatial calibration on the to-be-gripped target object and the related environment in the scene. The robot arm is guided to complete the gripping task according to trained networks.

First claim

What is claimed is:

1. A device for collaborative servo control of motion vision of a robot in an uncalibrated agricultural scene, comprising: a robot arm, a to-be-gripped target object, an image sensor and a control module, wherein
an end of the robot arm is provided with a mechanical gripper;
the to-be-gripped target object is within a grip range of the robot arm;
the control module is electrically connected to the robot arm and the image sensor, respectively;
the control module drives the mechanical gripper to grip the to-be-gripped target object, and controls the image sensor to perform image sampling on a process of gripping the to-be-gripped target object by the robot arm; and
the image sensor sends sampled image data to the control module, wherein the control module is configured to:
construct a scene space feature vector acquisition network for acquiring a scene feature, the scene space feature vector acquisition network being pre-trained; and
simulate gripping through a domain randomization algorithm in a simulation environment to create simulation data.

2. The device according to claim 1 , wherein the robot arm is a six-degree-of-freedom robot arm.

3. The device according to claim 1 , wherein the control module is further configured to:
use the simulation data to pre-train an inverse reinforcement reward policy network.

4. The device according to claim 3 , wherein the control module is further configured to:
based on the pre-trained networks, obtain a programming result through a guided policy search (GPS) algorithm.

5. A method for collaborative servo control of motion vision of a robot in an uncalibrated agricultural scene, comprising:
constructing a scene space feature vector acquisition network, and acquiring a scene space feature vector;
acquiring a demonstrated action sample;
constructing an inverse reinforcement reward policy network;
subjecting the inverse reinforcement reward policy network to a transfer training; and
acquiring, based on a visual feature extraction network and the inverse reinforcement reward policy network, a forward-guided programming result by using a guided policy search (GPS) algorithm.

6. The method according to claim 5 , wherein the scene space feature vector acquisition network is a vision-based convolutional neural network.

7. The method according to claim 3 , wherein the step of acquiring the scene space feature vector comprises:
performing, by an image sensor, image sampling on a process of gripping a to-be-gripped target object by a robot arm, and extracting red, green and blue (RGB) image information; and
inputting the RGB image information into the scene space feature vector acquisition network to output the scene space feature vector.

8. The method according to claim 5 , wherein the step of acquiring the demonstrated action sample comprises:
pulling a robot arm to complete gripping a to-be-gripped target object, and acquiring demonstrated gripping action data of a single demonstrated gripping;
driving the robot arm to simulate the demonstrated gripping action data and autonomously complete an action of gripping the to-be-gripped target object, and acquiring image feature data of a demonstrated gripping scene through shooting; and
integrating the demonstrated gripping action data and the image feature data of the demonstrated gripping scene to obtain the demonstrated action sample.

9. The method according to claim 5 , wherein the step of constructing the inverse reinforcement reward policy network comprises:
constructing the inverse reinforcement reward policy network for fitting and representing a reward;
generating a simulation parameter through a simulation domain randomization algorithm;
programming and simulating a virtual gripping action by using a robot operating system (ROS) programming library, and obtaining a simulated gripping path through sampling; and
subjecting the inverse reinforcement reward policy network to a simulation pre-training.

10. The method according to claim 5 , wherein the transfer training of the inverse reinforcement reward policy network comprises:
performing optimization training on the inverse reinforcement reward policy network by using the demonstrated action sample.

Assignees

• Univ Shanghai Jiaotong

Inventors

• Liu Chengliang
• Gong Liang
• Li Xudong
• Wang Tao
• Shen Xiaoye
• Lin Chenhui
• Tao Jianfeng

Classifications

• G05B2219/40269

  Naturally compliant robot arm · CPC title

• G05B19/4155

  characterised by program execution, i.e. part program or machine function execution, e.g. selection of a program · CPC title

• B25J13/08

  by means of sensing devices, e.g. viewing or touching devices · CPC title

• B25J9/1661 Primary

  characterised by task planning, object-oriented languages · CPC title

• B25J9/163

  learning, adaptive, model based, rule based expert control · CPC title