Method for detecting grasping position of robot in grasping object

Patent metadata

Field	Value
Publication number	US-11878433-B2
Application number	US-202017032399-A
Country	US
Kind code	B2
Filing date	Sep 25, 2020
Priority date	Dec 12, 2018
Publication date	Jan 23, 2024
Grant date	Jan 23, 2024

Abstract

A method for detecting a grasping position of a robot in grasping a target object includes: collecting a target RGB image and a target Depth image of the target object at different view angles; inputting each of the target RGB image to a target object segmentation network for calculation to obtain an RGB pixel region of the target object in the target RGB image and a Depth pixel region of the target object; inputting the RGB pixel region to an optimal grasping position generation network to obtain an optimal grasping position for grasping the target object; inputting the Depth pixel region of the target object and the optimal grasping position to a grasping position quality evaluation network to calculate a score of the optimal grasping position; and selecting an optimal grasping position corresponding to a highest score as a global optimal grasping position of the robot.

First claim

What is claimed is:

1. A method for detecting a grasping position of a robot in grasping a target object, comprising:
collecting at least one target RGB image and at least one target Depth image of the target object at different view angles, wherein pixel points in the at least one target RGB image have one-to-one corresponding pixel points in the at least one target Depth image;
inputting each of the at least one target RGB image to a target object segmentation network for calculation to obtain an RGB pixel region of the target object in the target RGB image and a Depth pixel region of the target object in the target Depth image;
inputting the RGB pixel region of the target object to an optimal grasping position generation network to obtain an optimal grasping position for grasping the target object;
inputting the Depth pixel region of the target object and the optimal grasping position to a grasping position quality evaluation network to calculate a score of the optimal grasping position; and
selecting an optimal grasping position corresponding to a highest score as a global optimal grasping position of the robot.

2. The method according to claim 1 , wherein
the target object segmentation network is a network trained based on a convolutional neural network model, wherein the target object segmentation network is trained by:
acquiring an RGB image containing the target object;
zooming the RGB image to a first predetermined resolution to obtain a first training set;
annotating a pixel region corresponding to the target object in the first training set;
inputting the first training set and the pixel region corresponding to the target object to the convolutional neural network model for training to obtain the target object segmentation network;
performing an overlap comparison between the RGB pixel region corresponding to the target object obtained by the target object segmentation network and the annotated pixel region corresponding to the target object; and
adjusting a weight of the target object segmentation network according to the overlap comparison result.

3. The method according to claim 1 , wherein the optimal grasping position generation network is a network trained based on a convolutional neural network model, and the optimal grasping position generation network is trained by:
zooming the RGB pixel region corresponding to the target object obtained based on the target object segmentation network to a second predetermined resolution to obtain a second training set;
marking optimal grasping position coordinates for an image in the second training set; and
taking the image in the second training set and the corresponding optimal grasping position coordinates as inputs, and training the inputs based on the convolutional neural network model to obtain the optimal grasping position generation network.

4. The method according to claim 1 , wherein the grasping position quality evaluation network is a network trained based on a convolutional neural network model, and the grasping position quality evaluation network is trained by:
acquiring a Depth image containing the target object;
zooming the Depth image to a third predetermined resolution to obtain a third training set;
randomly acquiring a pair of grasping positions on the Depth image in the third training set, and calculating a corresponding score by a predetermined scoring algorithm; and
taking the Depth image, the grasping positions and the score corresponding to the grasping positions as inputs, and training the inputs based on the convolutional neural network model to obtain the grasping position quality evaluation network.

5. The method according to claim 2 , wherein the first predetermined resolution is a 320×320 pixel.

6. The method according to claim 3 , wherein the second predetermined resolution is a 227×227 pixel.

7. The method according to claim 4 , wherein the third predetermined resolution is a 32×32 pixel.

8. A computing device, comprising: a processor, a memory, a communication interface and a communication bus; wherein the processor, the memory and the communication bus communicate with each other via the communication bus; and
the memory is configured to store at least one executable instruction, wherein the at least one executable instruction causes the processor to perform the steps of:
collecting at least one target RGB image and at least one target Depth image of the target object at different view angles, wherein pixel points in the at least one target RGB image have one-to-one corresponding pixel points in the at least one target Depth image;
inputting each of the at least one target RGB image to a target object segmentation network for calculation to obtain an RGB pixel region of the target object in the target RGB image and a Depth pixel region of the target object in the target Depth image;
inputting the RGB pixel region of the target object to an optimal grasping position generation network to obtain an optimal grasping position for grasping the target object;
inputting the Depth pixel region of the target object and the optimal grasping position to a grasping position quality evaluation network to calculate a score of the optimal grasping position; and
selecting an optimal grasping position corresponding to a highest score as a global optimal grasping position of the robot.

9. The computing device according to claim 8 , wherein the target object segmentation network is a network trained based on a convolutional neural network model, wherein the target object segmentation network is trained by:
acquiring an RGB image containing the target object;
zooming the RGB image to a first predetermined resolution to obtain a first training set;
annotating a pixel region corresponding to the target object in the first training set;
inputting the first training set and the pixel region corresponding to the target object to the convolutional neural network model for training to obtain the target object segmentation network;
performing an overlap comparison between the RGB pixel region corresponding to the target object obtained by the target object segmentation network and the annotated pixel region corresponding to the target object; and
adjusting a weight of the target object segmentation network according to the overlap comparison result.

10. The computing device according to claim 8 , wherein the optimal grasping position generation network is a network trained based on a convolutional neural network model, and the optimal grasping position generation network is trained by:
zooming the RGB pixel region corresponding to the target object obtained based on the target object segmentation network to a second predetermined resolution to obtain a second training set;
marking optimal grasping position coordinates for an image in the second training set; and
taking the image in the second training set and the corresponding optimal grasping position coordinates as inputs, and training the inputs based on the convolutional neural network model to obtain the optimal grasping position generation network.

11. The computing device according to claim 8 , wherein the grasping position quality evaluation network is a network trained based on a convolutional neural network model, and the grasping position quality evaluation network is trained by:
acquiring a Depth image containing the target object;
zooming the Depth image to a third predetermined resolution to obtain a third training set;
randomly acquiring a pair of grasping positions on the Depth image in the third training set, and calculating a corresponding score by a predetermined scoring algorithm; and
taking the Depth image, the grasp

Assignees

• Cloudminds Shenzhen Robotics Systems Co Ltd
• Cloudminds Robotics Co Ltd

Inventors

• Du Guoguang
• Wang Kai
• Lian Shiguo

Classifications

• B25J9/1697 Primary

  Vision controlled systems · CPC title

• B25J9/1669

  characterised by special application, e.g. multi-arm co-operation, assembly, grasping · CPC title

• G06T1/0014

  Image feed-back for automatic industrial control, e.g. robot with camera (robots B25J19/023) · CPC title

• G06T7/11 Primary

  Region-based segmentation · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title