Device and method for ascertaining the pose of an object

Patent metadata

Field	Value
Publication number	US-11941846-B2
Application number	US-202217677015-A
Country	US
Kind code	B2
Filing date	Feb 22, 2022
Priority date	Feb 26, 2021
Publication date	Mar 26, 2024
Grant date	Mar 26, 2024

Abstract

A method for ascertaining the pose of an object. The method includes recording a first and a second camera image of the object, ascertaining a correspondence between camera pixels of the camera images and vertices of a 3D model of the object, and ascertaining the pose of the object from a set of poses by minimizing, across the set of poses, a loss function, the loss function for a pose being provided by accumulation of distance measures between projections of the object in the pose onto the respective camera image plane and the corresponding pixels of the respective camera image.

First claim

What is claimed is:

1. A method for ascertaining a pose of an object, comprising the following steps:
ascertaining a 3D model of the object, the 3D model including a grid of vertices;
recording a first camera image of the object from a first perspective;
recording a second camera image of the object from a second perspective, which differs from the first perspective;
ascertaining a correspondence between camera pixels of the first camera image and vertices of the 3D model;
ascertaining a correspondence between camera pixels of the second camera image and vertices of the 3D model; and
ascertaining the pose of the object from a set of poses by minimizing, across the set of poses, a loss function, the loss function for each pose being provided by:
projecting vertices of the 3D model into a respective camera image plane according to the first perspective, in the event that the 3 D model is situated in the pose,
projecting vertices of the 3D model into a respective camera image plane according to the second perspective, in the event that the 3D model is situated in the pose,
calculating distance measures between the projections of the vertices projected according to the first perspective and the camera pixels of the first camera image, which are assigned to vertices projected according to the first perspective,
calculating distance measures between the projections of the vertices projected according to the second perspective and the camera pixels of the second camera image, which are assigned to the vertices projected according to the second perspective, and
accumulating the calculated distance measures to the loss function.

2. The method as recited in claim 1 , wherein the ascertainment of the correspondence between camera pixels of the first camera image and vertices of the 3D model and the ascertainment of the correspondence between camera pixels of the second camera image and vertices of the 3D model take place by mapping camera pixels of the first camera image and of the second camera image onto respective descriptor values and by assigning to the camera pixels vertices of the 3D model, to which the respective descriptor values are assigned.

3. The method as recited in claim 2 , wherein the mapping takes place using a machine learning model, which is trained using the 3 D model and the descriptor values assigned to the vertices of the 3D model.

4. The method as recited in claim 1 , further comprising:
recording a plurality of camera images of the object, each camera image of the plurality of camera images being recorded from one respective perspective each of a plurality of perspectives differing pairwise,
ascertaining, for each camera image of the plurality of camera images, a correspondence between camera pixels of the first camera image and vertices of the 3D model; and
ascertaining the pose of the object from a set of poses by minimizing, across the set of poses, a loss function, the loss function for each pose being provided by:
for each camera image of the plurality of camera images, projecting vertices of the 3D model into a respective camera image plane according to the respective perspective, in the event that the 3D model is situated in the pose,
for each camera image of the plurality of camera images, calculating distance measures between the projections of the vertices projected according to the respective perspective and the camera pixels of the camera image, which are assigned to the vertices projected according to the respective perspective, and
accumulating the calculated distance measures to the loss function.

5. A method for controlling a robot, comprising the following steps:
ascertaining the pose of an object, including:
ascertaining a 3D model of the object, the 3D model including a grid of vertices;
recording a first camera image of the object from a first perspective;
recording a second camera image of the object from a second perspective, which differs from the first perspective;
ascertaining a correspondence between camera pixels of the first camera image and vertices of the 3D model;
ascertaining a correspondence between camera pixels of the second camera image and vertices of the 3D model; and
ascertaining the pose of the object from a set of poses by minimizing, across the set of poses, a loss function, the loss function for each pose being provided by:
projecting vertices of the 3D model into a respective camera image plane according to the first perspective, in the event that the 3 D model is situated in the pose,
projecting vertices of the 3D model into a respective camera image plane according to the second perspective, in the event that the 3D model is situated in the pose,
calculating distance measures between the projections of the vertices projected according to the first perspective and the camera pixels of the first camera image, which are assigned to vertices projected according to the first perspective,
calculating distance measures between the projections of the vertices projected according to the second perspective and the camera pixels of the second camera image, which are assigned to the vertices projected according to the second perspective, and
accumulating the calculated distance measures to the loss function; and

controlling the robot as a function of the ascertained pose of the object.

6. The method as recited in claim 5 , wherein the determination of the pose of the object includes the determination of the position of a particular part of the object, and the control of the robot as a function of the ascertained pose of the object includes controlling an end effector of the robot to move to the position of the particular part of the object and to interact with the particular part of the object.

7. A software agent or hardware agent, comprising:
a camera configured to record camera images of an object; and
a control unit configured to ascertain a pose of the object, the control unit configured to:
ascertain a 3D model of the object, the 3D model including a grid of vertices;
record a first camera image of the object from a first perspective;
record a second camera image of the object from a second perspective, which differs from the first perspective;
ascertain a correspondence between camera pixels of the first camera image and vertices of the 3D model;
ascertain a correspondence between camera pixels of the second camera image and vertices of the 3D model; and
ascertain the pose of the object from a set of poses by minimizing, across the set of poses, a loss function, the loss function for each pose being provided by:
projecting vertices of the 3D model into a respective camera image plane according to the first perspective, in the event that the 3D model is situated in the pose,
projecting vertices of the 3D model into a respective camera image plane according to the second perspective, in the event that the 3D model is situated in the pose,
calculating distance measures between the projections of the vertices projected according to the first perspective and the camera pixels of the first camera image, which are assigned to vertices projected according to the first perspective,
calculating distance measures between the projections of the vertices projected according to the second perspective and the camera pixels of the second camera image, which are assigned to the vertices projected according to the second perspective, and
accumulating the calculated distance measures to the loss function.

8. The software agent or hardware agent as recited in claim 7 , wherein the software agent of the hardware agent is a robot.

9. The software agent or hardware agent as recited in claim 7 , further comprising at least one actuator, the co

Assignees

• Bosch Gmbh Robert

Inventors

• Spies Markus

Classifications

• G06T7/75 Primary

  involving models · CPC title

• G06T1/0014

  Image feed-back for automatic industrial control, e.g. robot with camera (robots B25J19/023) · CPC title

• G06T17/20

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T3/06 Primary

  Topological mapping of higher dimensional structures onto lower dimensional surfaces · CPC title