Method and apparatus with object information estimation and virtual object generation

Patent metadata

Field	Value
Publication number	US-12175696-B2
Application number	US-202117146895-A
Country	US
Kind code	B2
Filing date	Jan 12, 2021
Priority date	Jan 13, 2020
Publication date	Dec 24, 2024
Grant date	Dec 24, 2024

Abstract

An electronic device for estimating object information and generating a virtual object and a method of operating the electronic device are disclosed. The method includes obtaining an image, obtaining a class feature, a pose feature, and a relationship feature of an object included in the image, correcting each of the class feature, the pose feature, and the relationship feature using any combination of any two or more of the class feature, the pose feature, and the relationship feature of the object, and obtaining class information, pose information, and relationship information of the object based on the corrected class feature, the corrected pose feature, and the corrected relationship feature, respectively.

First claim

What is claimed is:

1. A method of operating an electronic device, comprising:
obtaining an image;
extracting a class feature, a pose feature, and a relationship feature of an object included in the image;
correcting each of the class feature, the pose feature, and the relationship feature using any combination of any two or more of the class feature, the pose feature, and the relationship feature of the object;
extracting class information, pose information, and relationship information of the object based on the corrected class feature, the corrected pose feature, and the corrected relationship feature, respectively; and
predicting virtual object information, using a rendering prediction network, based on at least one of the class information, the pose information, or the relationship information, where the predicted virtual object information represents a relationship between a virtual object and surrounding objects, including the object, in the image to render the virtual object on the image for interaction with the surrounding objects,
wherein the predicting of the virtual object information further comprises predicting virtual position information, virtual pose information, and virtual action information of the virtual object to be rendered on the image based on the class information, the pose information, and the relationship information of the object.

2. The method of claim 1 , wherein the correcting comprises:
correcting one of the class feature, the pose feature, and the relationship feature by applying a preset weight to each of the class feature, the pose feature, and the relationship feature of the object.

3. The method of claim 1 , wherein the extracting of the class feature, the pose feature, and the relationship feature comprises:
extracting the class feature, the pose feature, and the relationship feature from respective intermediate layers of sub-networks of a neural network respectively corresponding to the class feature, the pose feature, and the relationship feature.

4. The method of claim 3 , wherein the intermediate layers of the sub-networks are connected to one another, and the class feature, the pose feature, and the relationship feature are shared by the sub-networks that are different from one another.

5. The method of claim 1 , wherein the extracting of the class information, the pose information, and the relationship information of the object comprises:
when the corrected class feature, the corrected pose feature, and the corrected relationship feature are input to respective subsequent layers of respective intermediate layers of sub-networks of a neural network respectively corresponding to the corrected class feature, the corrected pose feature, and the corrected relationship feature, extracting the class information, the pose information, and the relationship information from respective output layers of the corresponding sub-networks.

6. The method of claim 1 ,
wherein the class information includes information as to which object is detected in the image,
the pose information includes information indicating a rotation angle of an object detected in the image, and
the relationship information includes action information associated with either one or both of an action of an object detected in the image and connection information associated with a connection with another object.

7. The method of claim 1 , wherein the predicting of the virtual object information further comprises:
adding the virtual object to the image based on the virtual position information, the virtual pose information, and the virtual action information.

8. The method of claim 7 , wherein the adding of the virtual object comprises:
when at least one of the virtual position information, the virtual pose information, or the virtual action information is a respective plurality of sets of predicted information, adding the virtual object to the image based on information selected by a user from among the respective plurality of sets of the predicted information.

9. The method of claim 7 ,
wherein the virtual position information includes information indicating a position at which the virtual object is available to be rendered in the image,
the virtual pose information includes information indicating a rotation angle of the virtual object, and
the virtual action information includes information indicating an action of the virtual object.

10. The method of claim 1 , wherein the image is a red-green-blue (RGB) depth (D) (RGB-D) image.

11. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, configure the one or more processors to perform the method of claim 1 .

12. An electronic device comprising:
one or more processors configured to:
obtain an image;
extract a class feature, a pose feature, and a relationship feature of at least one object included in the image;
correct each of the class feature, the pose feature, and the relationship feature using any combination of any two or more of the class feature, the pose feature, and the relationship feature of the at least one object; and
extract class information, pose information, and relationship information of the at least one object based on the corrected class feature, the corrected pose feature, and the corrected relationship feature, respectively; and
predict virtual object information, using a rendering prediction network, based on at least one of the class information, the pose information, or the relationship information, where the predicted virtual object information represents a relationship between a virtual object and surrounding objects, including the object, in the image to render the virtual object on the image for interaction with the surrounding objects,
wherein, for the prediction of the virtual object information, the one or more processors are configured to predict virtual position information, virtual pose information, and virtual action information of the virtual object to be rendered on the image based on the class information, the pose information, and the relationship information of the object.

13. The electronic device of claim 12 , wherein, for the correction, the one or more processors are configured to:
correct one of the class feature, the pose feature, and the relationship feature by applying a preset weight to each of the class feature, the pose feature, and the relationship feature of the object.

14. The electronic device of claim 12 , wherein, for the extraction of the class feature, the pose feature, and the relationship feature, the one or more processors are configured to:
extract the class feature, the pose feature, and the relationship feature from respective intermediate layers of sub-networks of a neural network respectively corresponding to the class feature, the pose feature, and the relationship feature.

15. The electronic device of claim 14 , wherein the intermediate layers of the sub-networks are connected to one another, and the class feature, the pose feature, and the relationship feature are shared by the sub-networks that are different from one another.

16. The electronic device of claim 12 , wherein, for the extraction of the class information, the pose information, and the relationship information, the one or more processors are configured to:
when the corrected class feature, the corrected pose feature, and the corrected relationship feature are input to respective subsequent layers of respective intermediate layers of sub-networks of a neural network respectively corresponding to the corrected class feature

Assignees

• Samsung Electronics Co Ltd

Inventors

• Zhang Chao
• Li Weiming
• Wang Qiang
• Hong Sunghoon
• Kim Wooshik

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06V20/10

  Terrestrial scenes (scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59) · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title