Device and method for controlling a robot to pick up an object in various positions

Patent metadata

Field	Value
Publication number	US-11964400-B2
Application number	US-202117453920-A
Country	US
Kind code	B2
Filing date	Nov 8, 2021
Priority date	Nov 13, 2020
Publication date	Apr 23, 2024
Grant date	Apr 23, 2024

Abstract

A method for controlling a robot to pick up an object in various positions. The method includes: defining a plurality of reference points on the object; mapping a first camera image of the object in a known position onto a first descriptor image; identifying the descriptors of the reference points from the first descriptor image; mapping a second camera image of the object in an unknown position onto a second descriptor image; searching the identified descriptors of the reference points in the second descriptor image; ascertaining the positions of the reference points in the three-dimensional space in the unknown position from the found positions; and ascertaining a pickup pose of the object for the unknown position from the ascertained positions of the reference points.

First claim

What is claimed is:

1. A method for controlling a robot to pick up an object in various positions, the method comprising the following steps:
determining a plurality of reference points on the object, so that positions of the reference points in three-dimensional space define a pickup pose of the object;
receiving a first camera image of the object in a known position of the object, so that the positions of the reference points in the first camera image are known;
mapping the first camera image onto a first descriptor image using a machine learning model that is trained to assign object points visible in camera images to descriptors, independent of positions of the visible object points in the camera images;
identifying descriptors of the reference points from the first descriptor image by reading out the first descriptor image at the known positions of the reference points;
receiving a second camera image of the object in an unknown position in which the object is to be picked up;
mapping the second camera image onto a second descriptor image using the machine learning model;
searching for the identified descriptors of the reference points in the second descriptor image;
ascertaining the positions of the reference points in the three-dimensional space in the unknown position from found positions of the descriptors of the reference points in the second descriptor image;
ascertaining a pickup pose of the object for the unknown position from the ascertained positions of the reference points in the three-dimensional space; and
controlling the robot device to pick up the object corresponding to the ascertained pickup pose.

2. The method as recited in claim 1 , wherein the positions of the reference points in the three-dimensional space in the unknown position are ascertained from the found positions of the descriptors of the reference points in the second descriptor image, by projecting depth information for the positions of the reference points in the second camera image, corresponding to the positions of the reference points in the second descriptor image, into the three-dimensional space.

3. The method as recited in claim 1 , wherein the reference points include at least two reference points that are defined in such a way that they are situated along an extended segment of the object, and the pickup pose of the object for the unknown position being ascertained by ascertaining an axis that extends through the at least two reference points.

4. The method as recited in claim 1 , wherein the reference points include at least three reference points that are defined in such a way that they are situated on a flat surface of the object, and the pickup pose of the object for the unknown position being ascertained by ascertaining a plane that extends through the at least three reference points.

5. The method as recited in claim 1 , wherein training of the machine learning model includes:
obtaining a 3D model of the object, the 3D model including a grid of vertices;
determining a descriptor for each vertex of the grid;
producing training data image pairs, each training data image pair including a training input image that indicates the object and a target image, and producing of the target image includes:
determining the vertex positions of vertices of the object model of the object that the vertices have in the training input image, and
assigning, for each determined vertex position in the training input image, the descriptor determined for the vertex at the vertex position to the position in the target image; and

training the machine learning model through supervised learning, using the training data image pairs as training data.

6. The method as recited in claim 5 , wherein the producing of the training data image pairs includes obtaining a plurality of images of the object in various positions and producing a training data image pair from each obtained image by producing a respective target image for the obtained image.

7. The method as recited in claim 5 , further comprising:
determining the vertex positions of vertices of the object model of the object that the vertices have in the training input images from the respective positions that the object has in the training input images.

8. The method as recited in claim 5 , wherein the vertices of the 3D model are connected by edges, each edge of the edges having a weight that specifies a closeness of two vertices in the object connected by the edge, and the determining of the descriptor for each vertex of the grid takes place through a search of descriptors for the vertices that minimize a sum, over pairs of connected vertices, of distances between the descriptors of the pair of vertices, weighted by the weight of the edge between the pair of vertices.

9. A robot control device configured to control a robot to pick up an object in various positions, the robot control device configured to:
determine a plurality of reference points on the object, so that positions of the reference points in three-dimensional space define a pickup pose of the object;
receive a first camera image of the object in a known position of the object, so that the positions of the reference points in the first camera image are known;
map the first camera image onto a first descriptor image using a machine learning model that is trained to assign object points visible in camera images to descriptors, independent of positions of the visible object points in the camera images;
identify descriptors of the reference points from the first descriptor image by reading out the first descriptor image at the known positions of the reference points;
receive a second camera image of the object in an unknown position in which the object is to be picked up;
map the second camera image onto a second descriptor image using the machine learning model;
search for the identified descriptors of the reference points in the second descriptor image;
ascertain the positions of the reference points in three-dimensional space in the unknown position from found positions of the descriptors of the reference points in the second descriptor image;
ascertain a pickup pose of the object for the unknown position from the ascertained positions of the reference points in the three-dimensional space; and
control the robot device to pick up the object corresponding to the ascertained pickup pose.

10. A non-transitory computer-readable medium on which is stored a computer program including instructions for controlling a robot to pick up an object in various positions, the instructions, when executed by a processor, causing the processor to perform the following steps:
determining a plurality of reference points on the object, so that positions of the reference points in three-dimensional space define a pickup pose of the object;
receiving a first camera image of the object in a known position of the object, so that the positions of the reference points in the first camera image are known;
mapping the first camera image onto a first descriptor image using a machine learning model that is trained to assign object points visible in camera images to descriptors, independent of positions of the visible object points in the camera images;
identifying descriptors of the reference points from the first descriptor image by reading out the first descriptor image at the known positions of the reference points;
receiving a second camera image of the object in an unknown position in which the object is to be picked up;
mapping the second camera image onto a second descriptor image using the machine learning model;
searching for the identified descriptors of the reference points in the second descriptor image;
ascertai

Assignees

• Bosch Gmbh Robert

Inventors

• Kupcsik Andras Gabor
• Todescato Marco
• Spies Markus
• Waniek Nicolai
• Schillinger Philipp Christian
• Buerger Mathias

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• B25J9/1697 Primary

  Vision controlled systems · CPC title

• B25J9/163

  learning, adaptive, model based, rule based expert control · CPC title

• B25J9/1664

  characterised by motion, path, trajectory planning · CPC title