Method and apparatus with object detection

Patent metadata

Field	Value
Publication number	US-12217518-B2
Application number	US-202117529634-A
Country	US
Kind code	B2
Filing date	Nov 18, 2021
Priority date	Nov 26, 2020
Publication date	Feb 4, 2025
Grant date	Feb 4, 2025

Abstract

A method with object detection includes obtaining a first point cloud feature based on point cloud data of an image and determining at least one object in the image based on the first point cloud feature.

First claim

What is claimed is:

1. A processor-implemented method for object identification based on point cloud features extracted from point cloud data corresponding to an image, the method comprising:
generating first appearance features and first geometric features, both of respective objects of the image, dependent on the extracted point cloud features and correspondingly determined geometric centers of the respective objects; and
identifying the respective objects by generating corresponding second appearance features and second geometric features based on:
the extracted point cloud features of an object from the respective objects; and
appearance relationship weightings and geometric relationship weightings, which are respectively based on the generated first appearance features and the generated first geometric features,

wherein the second appearance features that correspond to the object are determined based on an application of a relationship weighting to the extracted point cloud features of the object, with the relationship weighting being based on both a corresponding one of the appearance relationship weightings and a corresponding one of the geometric relationship weightings; and
controlling an operation of an augmented reality system based on the identified respective objects.

2. The method of claim 1 , further comprising:
performing point cloud complement on first point cloud data of the image to generate the point cloud data corresponding to the image; and
performing the extracting of point cloud features from the generated point cloud data corresponding to the image.

3. The method of claim 2 , wherein the performing of the point cloud complement on the point cloud data of the image comprises:
extracting a high-dimensional feature of the first point cloud data of the image; and
generating the complemented point cloud data by complementing a point cloud based on the high-dimensional feature.

4. A method with object detection, the method comprising:
extracting respective first point cloud features of respective objects in an image from point cloud data corresponding to the image;
generating first appearance features and first geometric features of respective objects in the image based on the respective first point cloud features;
generating appearance relationship weightings and geometric relationship weightings between candidate regions corresponding to the respective objects in the image, based on the first appearance features and the first geometric features of the respective objects in the image;
generating second appearance features and second geometric features of the respective objects in the image, based on the respective first point cloud features, the appearance relationship weightings, and the geometric relationship weightings;
determining one object from the respective objects in the image, based on the second appearance features and the second geometric features corresponding to the one object; and
controlling an operation of an augmented reality system based on the determined at least one object,
wherein the obtaining of the second appearance features of the one object includes applying a relationship weight corresponding to the one object to the respective first point cloud features of the one object, where the relationship weight is based on the appearance relationship weighting corresponding to the one object and the geometric relationship weighting corresponding to the one object.

5. The method of claim 4 , wherein the obtaining of the geometric weightings between the candidate regions comprises:
obtaining a geometric feature-related vector between the candidate regions based on the first geometric features of the respective objects in the image; and
obtaining the geometric weightings between the candidate regions based on the geometric feature-related vector between the candidate regions.

6. The method of claim 5 , wherein the first geometric features, the second geometric features, and the geometric feature-related vector are related to any one or any combination of any two or more of center positions, sizes, and orientations of the objects.

7. The method of claim 1 , further comprising rendering a virtual object in an augmented reality system, based on one of the identified respective objects.

8. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 1 .

9. An apparatus for object identification based on point cloud features extracted from point cloud data corresponding to an image, the apparatus comprising:
one or more processors configured to:
generate first appearance features and first geometric features, both of respective objects of the image, dependent on the extracted point cloud features and correspondingly determined geometric centers of the respective objects; and
identify the respective objects by generating corresponding second appearance features and second geometric features based on:
the extracted point cloud features of an object from the respective objects; and
appearance relationship weightings and geometric relationship weightings, which are respectively based on the generated first appearance features and the generated first geometric features,

wherein the second appearance features that corresponds to the object are determined based on an application of a relationship weighting to the extracted point cloud features of the object, with the relationship weighting being based on both a corresponding one of the appearance relationship weightings and a corresponding one of geometric relationship weightings; and
control an operation of an augmented reality system based on the identified respective objects.

10. The apparatus of claim 9 , wherein the one or more processors are configured to:
perform a point cloud complement on the first point cloud data of the image to generate the point cloud data corresponding to the image; and
performing the extracting of point cloud features from the generated point cloud data corresponding to the image.

11. The apparatus of claim 10 , wherein, for the performing of the point cloud complement, the one or more processors are configured to extract a high-dimensional feature of the first point cloud data of the image and generate the complemented point cloud data by complementing a point cloud based on the high-dimensional feature.

12. The apparatus of claim 9 , wherein, for the generating of the corresponding second appearance features and second geometric features, the one or more processors are configured to:
generate the appearance relationship weightings and the geometric relationship weightings; and
apply a relationship weight, generated based on the appearance relationship weightings and the geometric relationship weightings, to the extracted point cloud features of the object.

13. The apparatus of claim 12 , wherein, for the generating of the geometric relationship weightings, the one or more processors are configured to:
generate a geometric feature-related vector between candidate regions corresponding to the respective objects dependent on the determined geometric centers; and
generate the geometric relationship weightings based on the geometric feature-related vector between the candidate regions.

14. The apparatus of claim 13 , wherein the first geometric features, the second geometric features, and the geometric feature-related vector are related to any one or any combination of any two or more of center positions, sizes, and orientations of the objects.

15. An electronic device, compris

Assignees

• Samsung Electronics Co Ltd

Inventors

• Li Weiming
• Kim Jiyeon
• Wang Qiang
• Chang Hyun Sung
• Hong Sunghoon
• Zhang Chao

Classifications

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06V20/20

  in augmented reality scenes · CPC title

• G06V20/64 Primary

  Three-dimensional [3D] objects · CPC title