Method and apparatus with object pose estimation

Patent metadata

Field	Value
Publication number	US-12175698-B2
Application number	US-202117545333-A
Country	US
Kind code	B2
Filing date	Dec 8, 2021
Priority date	Dec 8, 2020
Publication date	Dec 24, 2024
Grant date	Dec 24, 2024

Abstract

A processor-implemented method with object pose estimation includes: determining an image feature corresponding to a point cloud of an input image; determining semantic segmentation information, instance mask information, and keypoint information of an object, based on the image feature; and estimating a pose of the object based on the semantic segmentation information, the instance mask information, and the keypoint information.

First claim

What is claimed is:

1. A processor-implemented method with object pose estimation, the method comprising:
determining an image feature corresponding to a point cloud of an input image;
determining semantic segmentation information, instance mask information, and keypoint information of an object, based on the image feature; and
estimating a pose of the object based on the semantic segmentation information, the instance mask information, and the keypoint information.

2. The method of claim 1 , wherein the input image comprises a depth image, and wherein the determining of the image feature comprises: extracting a point cloud feature based on the depth image; and determining the point cloud feature to be the image feature.

3. The method of claim 2 , wherein the extracting of the point cloud feature comprises:
determining point cloud information corresponding to the depth image; and
extracting the point cloud feature based on any one or any combination of any two or more of the point cloud information, a color feature, and a normal feature.

4. The method of claim 2 ,
wherein the input image further includes either one or both of a color image and a grayscale image, and
wherein the determining of the image feature comprises:
extracting a first image feature based on either one or both of the color image and the grayscale image; and
determining the image feature by fusing the point cloud feature and the first image feature.

5. The method of claim 4 , wherein the determining of the image feature by fusing the point cloud feature and the first image feature comprises determining the image feature by pixel-wise fusing the point cloud feature and the first image feature.

6. The method of claim 1 , wherein the determining of the semantic segmentation information, the instance mask information, and the keypoint information based on the image feature comprises:
determining the semantic segmentation information corresponding to the point cloud based on the image feature;
generating a three-dimensional (3D) grid based on point cloud information corresponding to the input image and determining the instance mask information based on the 3D grid; and
determining the keypoint information using the instance mask information or using the semantic segmentation information and the instance mask information.

7. The method of claim 6 ,
wherein the instance mask information indicates grid information corresponding to the point cloud in the 3D grid, and
wherein network information corresponding to each point cloud of the object is determined based on grid information corresponding to a point cloud of a center of the object.

8. The method of claim 6 , wherein the generating of the 3D grid based on the point cloud information comprises any one or any combination of any two or more of:
determining a 3D grid by dividing a 3D space corresponding to the point cloud information at equal intervals;
determining multiple 3D grids by dividing the 3D space corresponding to the point cloud information into different intervals; and
determining multiple 3D grids by dividing the 3D space corresponding to the point cloud information based on same division starting points of different intervals.

9. The method of claim 6 , wherein the determining of the keypoint information using the instance mask information comprises either one or both of:
estimating a first offset of a keypoint corresponding to each point cloud based on the image feature and determining the keypoint information through regression based on the first offset and the instance mask information; and
estimating a second offset of a keypoint corresponding to each point cloud in each cell of the 3D grid based on the image feature and the instance mask information, and determining the keypoint information through regression based on the second offset.

10. The method of claim 6 , wherein the determining of the keypoint information using the instance mask information comprises:
estimating a second offset of a keypoint corresponding to each point cloud in each cell of the 3D grid based on the image feature and the instance mask information; and
determining the keypoint information through regression based on the second offset by determining a target predicted value of a key point predicted based on a point cloud, based on the second offset and the point cloud information, and determining the keypoint information through regression based on the target predicted value.

11. The method of claim 10 , wherein the determining of the target predicted value and the determining of the keypoint information through the regression based on the target predicted value comprises any one or any combination of any two or more of:
for each keypoint of the object, determining an average value of a target predicted value corresponding to a keypoint and each point cloud, as keypoint information;
for each keypoint of the object, determining a weighted average value of a probability value corresponding to each point cloud in the instance mask information and a target predicted value corresponding to a keypoint and each point cloud, as keypoint information;
for each keypoint of the object, determining a weighted average value of a target predicted value corresponding to a keypoint and a point cloud of a preset value closest to a central point of the object and a probability value corresponding to the point cloud of the preset value in the instance mask information, as keypoint information of the object; and
for each keypoint of the object, determining a weighted average value of a distance approximate value corresponding to a keypoint and each point cloud, a target predicted value corresponding to a keypoint and each point cloud, and a probability value corresponding to each point cloud in the instance mask information, as keypoint information.

12. The method of claim 6 , wherein the determining of the keypoint information using the semantic segmentation information and the instance mask information comprises:
determining instance segmentation information based on the semantic segmentation information and the instance mask information;
estimating a first offset of a keypoint corresponding to each point cloud based on the image feature; and
determining the keypoint information through regression based on the first offset and the instance segmentation information.

13. The method of claim 12 , wherein the determining of the keypoint information through the regression based on the first offset and the instance segmentation information comprises:
determining an initial predicted value of a keypoint predicted based on a point cloud, based on the first offset and the point cloud information;
determining a target predicted value of a keypoint in the 3D grid based on the initial predicted value and the instance mask information; and
determining the keypoint information through regression based on the target predicted value.

14. The method of claim 12 , wherein the determining of the keypoint information through the regression based on the first offset and the instance segmentation information comprises:
determining an initial predicted value of a keypoint predicted based on a point cloud, based on the first offset and the point cloud information;
determining a target predicted value of a keypoint in the 3D grid based on the initial predicted value and the instance segmentation information; and
determining keypoint information of an object through a regression scheme based on the target predicted value.

15. A non-transitory computer-readable storage medium storing instructions that, when e

Assignees

• Samsung Electronics Co Ltd

Inventors

• Li Weiming
• Kim Jiyeon
• Chang Hyun Sung
• Wang Qiang
• Hong Sunghoon
• Liu Yang
• Wang Hao
• Kao Yueying

Classifications

• G06T7/50

  Depth or shape recovery · CPC title

• G06T2207/20021

  Dividing image into blocks, subimages or windows · CPC title

• G06V10/25

  Determination of region of interest [ROI] or a volume of interest [VOI] · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06T7/10

  Segmentation; Edge detection (motion-based segmentation G06T7/215) · CPC title