Method and system to generate a 3D model for a robot scene

Patent metadata

Field	Value
Publication number	US-11407111-B2
Application number	US-201816019768-A
Country	US
Kind code	B2
Filing date	Jun 27, 2018
Priority date	Jun 27, 2018
Publication date	Aug 9, 2022
Grant date	Aug 9, 2022

Abstract

A robot is configured to perform a task on an object using a method for generating a 3D model sufficient to determine a collision free path and identify the object in an industrial scene. The method includes determining a predefined collision free path and scanning an industrial scene around the robot. Stored images of the industrial scene are retrieved from a memory and analyzed to construct a new 3D model. After an object is detected in the new 3D model, the robot can further scan the image in the industrial scene while moving along a collision free path until the object is identified at a predefined certainty level. The robot can then perform a robot task on the object.

First claim

What is claimed is:

1. A method comprising:
determining a predefined collision free robot path;
moving a robot along the predefined collision free robot path;
scanning an industrial scene with a scanning sensor mounted on the robot while moving along the predefined collision free robot path;
storing scanned images of the industrial scene in a memory;
constructing a 3D model of the industrial scene based on the images stored in the memory;
performing a 3D object detection and recognition analysis of the 3D model;
based on the 3D object detection and recognition analysis failing to meet a detection and analysis threshold, updating the 3D model by moving the robot along one or more first new collision free robot paths and scanning the industrial scene with the scanning sensor based on moving the robot along the one or more first new collision free robot paths, wherein the detection and analysis threshold indicates whether the 3D model is sufficient to detect and recognize an object;
based on the 3D object detection and recognition analysis meeting the detection and analysis threshold, performing an object grasping and position analysis of the 3D model;
based on the object grasping and position analysis failing to meet a grasping and position analysis threshold, updating the 3D model by moving the robot along one or more second new collision free robot paths and scanning the industrial scene with the scanning sensor based on moving the robot along the one second new collision free robot paths, wherein the grasping and position analysis threshold indicates whether the 3D model is sufficient to direct the robot to grasp the object; and
based on the object grasping and position analysis meeting the grasping and position analysis threshold, directing the robot to grasp and pick up the object.

2. The method of claim 1 , wherein the scanning sensor is a 3D camera.

3. The method of claim 1 further comprising moving the scanning sensor relative to the robot when determining the predefined collision free path.

4. The method of claim 3 , wherein the moving of the scanning sensor includes panning, tilting, rotating and translating movement relative to the robot.

5. The method of claim 3 , wherein the moving of the scanning sensor includes moving an arm of the robot while a base of the robot remains stationary.

6. The method of claim 1 further comprising planning the collision free path with a controller having a collision free motion planning algorithm.

7. The method of claim 1 , wherein the planning of the collision free path occurs in real time without offline computer analysis.

8. A control system comprising:
a controller configured to:
determine a predefined collision free robot path;
move a robot along the predefined collision free robot path;
scan an industrial scene with a scanning sensor mounted on the robot while moving along the predefined collision free robot path to obtain first scanned images;
store scanned images of the industrial scene in a memory;
construct a 3D model of the industrial scene based on the scanned images;
perform a 3D object detection and recognition analysis of the 3D model;
based on the 3D object detection and recognition analysis of the 3D model failing to meet a detection and analysis threshold, update the 3D model by moving the robot along one or more first new collision free robot paths and scanning the industrial scene with the scanning sensor based on moving the robot along the one or more first new collision free robot paths, wherein the detection and analysis threshold indicates whether the 3D model is sufficient to detect and recognize an object;
based on the 3D object detection and recognition analysis of the 3D model meeting the detection and analysis threshold, perform an object grasping and position analysis of the 3D model;
based on the object grasping and position analysis failing to meet a grasping and position analysis threshold, update the 3D model by moving the robot along one or more second new collision free robot paths and scanning the industrial scene with the scanning sensor based on moving the robot along the one second new collision free robot paths, wherein the grasping and position analysis threshold indicates whether the 3D model is sufficient to direct the robot to grasp the object; and
based on the object grasping and position analysis meeting the grasping and position analysis threshold, direct the robot to grasp and pick up the object.

Assignees

• Abb Schweiz Ag

Inventors

• Zhang Biao
• Boca Remus
• Morato Carlos W
• Martinez Carlos
• Wang Jianjun
• Teng Zhou
• Huang Jinmiao
• Wahlstrom Magnus
• Holmberg Johnny

Classifications

• B25J9/1671

  characterised by simulation, either to verify existing program or to create and verify new program, CAD/CAM oriented, graphic oriented programming systems · CPC title

• B25J9/1676

  Avoiding collision or forbidden zones · CPC title

• B25J19/023

  including video camera means · CPC title

• G05B2219/40424

  Online motion planning, in real time, use vision to detect workspace changes · CPC title

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title