Method for controlling a robot for manipulating, in particular picking up, an object

Patent metadata

Field	Value
Publication number	US-12350847-B2
Application number	US-202318334959-A
Country	US
Kind code	B2
Filing date	Jun 14, 2023
Priority date	Jun 22, 2022
Publication date	Jul 8, 2025
Grant date	Jul 8, 2025

Abstract

A method for controlling a robot for manipulating, in particular picking up, an object. The method includes: creating an image which depicts the object; generating a manipulation-quality image from the image, in which, for each pixel which represents a point on the surface of the object, the pixel value of the pixel provides an assessment of how well the object may be manipulated at the point; recording descriptors of points of the object which should be used during the manipulation and/or of points which should be avoided during the manipulation; mapping the image onto a descriptor image; generating a manipulation-preference image by comparing the recorded descriptors of points to the descriptor image; selecting a point for manipulating the object taking into account the pixel values of the manipulation-quality image and the pixel values of the manipulation-preference image; and controlling the robot to manipulate the object at the selected point.

First claim

What is claimed is:

1. A method for controlling a robot for manipulating, including picking up, an object, comprising the following steps:
creating an image which depicts the object;
generating a manipulation-quality image from the image, in which, for each pixel which represents a point on a surface of the object, the pixel value of the pixel provides an assessment of how well the object may be manipulated at the point;
recording descriptors: (i) of points of the object which, according to a user input, should be used during the manipulation, and/or (ii) of points which, according to a user input, should be avoided during the manipulation;
mapping the image onto a descriptor image;
generating a manipulation-preference image by comparing the recorded descriptors of points to the descriptor image, wherein in the manipulation-preference image, for each pixel which represents a point on a surface of the object, a pixel value of the pixel provides an assessment of whether the object should be manipulated at the point;
selecting a point for manipulating the object taking into account the pixel values of the manipulation-quality image and the pixel values of the manipulation-preference image; and
controlling the robot to manipulate the object at the selected point.

2. The method as recited in claim 1 , wherein: (i) the manipulation-quality image is mapped by a first neural network, which is trained to map images of objects and/or information derived from images of objects onto manipulation-quality images, and/or (ii) the image is mapped onto the descriptor image by a second neural network, which is trained to map images of objects onto descriptor images.

3. The method as recited in claim 1 , further comprising:
combining the manipulation-quality image and the manipulation-preference image, and selecting the point for manipulating the object using the combination.

4. The method as recited in claim 1 , wherein the assessment for whether the object should be manipulated at the point is the degree of correlation between the descriptor which is assigned to the point in the descriptor image and one of the recorded descriptors.

5. The method as recited in claim 1 , further comprising:
generating the manipulation-quality image by forming a descriptor-correlation image for each recorded descriptor, in which, for each pixel which represents a point on the surface of the object, a pixel value of the pixel indicates how well the descriptor which is assigned to the point in the descriptor image correlates with the recorded descriptor, and combining the descriptor-correlation images to form the manipulation-quality image.

6. The method as recited in claim 5 , further comprising:
combining the manipulation-quality image and the manipulation-preference image through pixel-wise multiplication, calculating the pixel-wise maximum, calculating the pixel-wise minimum, excluding points for which the manipulation-quality image indicates a manipulation quality below a predetermined minimum quality, excluding points for which the manipulation-preference image indicates a correlation with a descriptor below a minimum correlation for a point which, according to the user input, should be used during the manipulation, and/or excluding points for which the manipulation-preference image indicates a correlation with a descriptor above a predetermined maximum correlation, recorded for a point which, according to the user input, should be avoided during the manipulation.

7. A robot control device configured to control a robot for manipulating, including picking up, an object, the robot control device configured to:
create an image which depicts the object;
generate a manipulation-quality image from the image, in which, for each pixel which represents a point on a surface of the object, the pixel value of the pixel provides an assessment of how well the object may be manipulated at the point;
record descriptors: (i) of points of the object which, according to a user input, should be used during the manipulation, and/or (ii) of points which, according to a user input, should be avoided during the manipulation;
map the image onto a descriptor image;
generate a manipulation-preference image by comparing the recorded descriptors of points to the descriptor image, wherein in the manipulation-preference image, for each pixel which represents a point on a surface of the object, a pixel value of the pixel provides an assessment of whether the object should be manipulated at the point;
select a point for manipulating the object taking into account the pixel values of the manipulation-quality image and the pixel values of the manipulation-preference image; and
control the robot to manipulate the object at the selected point.

8. A non-transitory computer-readable medium on which are stored commands for controlling a robot for manipulating, including picking up, an object, the commands, when executed by a processor, causing the processor to perform the following steps:
creating an image which depicts the object;
generating a manipulation-quality image from the image, in which, for each pixel which represents a point on a surface of the object, the pixel value of the pixel provides an assessment of how well the object may be manipulated at the point;
recording descriptors: (i) of points of the object which, according to a user input, should be used during the manipulation, and/or (ii) of points which, according to a user input, should be avoided during the manipulation;
mapping the image onto a descriptor image;
generating a manipulation-preference image by comparing the recorded descriptors of points to the descriptor image, wherein in the manipulation-preference image, for each pixel which represents a point on a surface of the object, a pixel value of the pixel provides an assessment of whether the object should be manipulated at the point;
selecting a point for manipulating the object taking into account the pixel values of the manipulation-quality image and the pixel values of the manipulation-preference image; and
controlling the robot to manipulate the object at the selected point.

Assignees

• Bosch Gmbh Robert

Inventors

• Kupcsik Andras Gabor
• Graf Christian
• Gabriel Miroslav
• Schillinger Philipp Christian

Classifications

• G06T7/0004

  Industrial image inspection · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20212

  Image combination · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T11/00 Primary

  Two-dimensional [2D] image generation · CPC title