Pose estimation and model retrieval for objects in images

Patent metadata

Field	Value
Publication number	US-2019147221-A1
Application number	US-201815946669-A
Country	US
Kind code	A1
Filing date	Apr 5, 2018
Priority date	Nov 15, 2017
Publication date	May 16, 2019
Grant date	—

Abstract

Techniques are provided for selecting a three-dimensional model. An input image including an object can be obtained, and a pose of the object in the input image can be determined. One or more candidate three-dimensional models representing one or more objects in the determined pose can be obtained. From the one or more candidate three-dimensional models, a candidate three-dimensional model can be determined to represent the object in the input image.

First claim

What is claimed is:

1 . A method of selecting a three-dimensional model, the method comprising:
obtaining an input image including an object; determining a pose of the object in the input image; obtaining one or more candidate three-dimensional models representing one or more objects in the determined pose; and determining, from the one or more candidate three-dimensional models, a candidate three-dimensional model to represent the object in the input image.

2 . The method of claim 1 , further comprising generating an output image based on the candidate three-dimensional model and the input image.

3 . The method of claim 1 , further comprising:
receiving a user input to manipulate the candidate three-dimensional model; and adjusting one or more of a pose or a location of the candidate three-dimensional model in an output image based on the user input.

4 . The method of claim 1 , further comprising:
obtaining an additional input image, the additional input image including the object in one or more of a different pose or a different location than a pose or location of the object in the input image; and adjusting one or more of a pose or a location of the candidate three-dimensional model in an output image based on a difference between the pose or location of the object in the additional input image and the pose or location of the object in the input image.

5 . The method of claim 1 , wherein obtaining the one or more three-dimensional models representing the one or more objects includes:
obtaining a plurality of three-dimensional models representing a plurality of objects; and selecting a subset of the plurality of three-dimensional models as the one or more candidate three-dimensional models.

6 . The method of claim 5 , further comprising:
determining a category of the object in the input image; and determining one or more categories associated with the plurality of candidate three-dimensional models; wherein the one or more candidate three-dimensional models are selected from the plurality of candidate three-dimensional models based on the one or more candidate three-dimensional models having the category of the object in the input image.

7 . The method of claim 1 , further comprising:
generating one or more images for the one or more candidate three-dimensional models, wherein the one or more images are generated to include the one or more objects in the determined pose; generating a descriptor for the object in the input image; and generating one or more descriptors for the one or more images generated for the one or more candidate three-dimensional models, wherein the candidate three-dimensional model is determined based on the descriptor generated for the object and the one or more descriptors generated for the one or more images.

8 . The method of claim 7 , further comprising:
comparing the one or more descriptors generated for the one or more images to the descriptor generated for the input image; and determining a descriptor from the one or more descriptors generated for the one or more images that has a closest match to the descriptor generated for the input image, wherein the candidate three-dimensional model is determined based on the descriptor from the one or more descriptors having the closest match to the descriptor generated for the input image.

9 . The method of claim 8 , wherein comparing the one or more descriptors generated for the one or more images to the descriptor generated for the input image includes:
performing a nearest-neighbor search using the descriptor generated for the input image as input.

10 . The method of claim 7 , wherein the one or more images generated for the one or more candidate three-dimensional models include one or more depth maps.

11 . The method of claim 1 , wherein the one or more candidate three-dimensional models include one or more three-dimensional meshes representing the one or more objects.

12 . The method of claim 1 , wherein determining the pose of the object in the input image includes:
determining a plurality of two-dimensional projections of a three-dimensional bounding box of the object in the input image; and estimating the pose of the object using the plurality of two-dimensional projections of the three-dimensional bounding box.

13 . The method of claim 12 , wherein the plurality of two-dimensional projections of the three-dimensional bounding box are determined by applying a trained convolutional network to the image, wherein the trained convolutional network is trained to predict two-dimensional projections of the three-dimensional bounding box of the object in a plurality of poses.

14 . The method of claim 12 , wherein estimating the pose of the object using the plurality of two-dimensional projections of the three-dimensional bounding box includes applying a perspective-n-point (PnP) problem using correspondences between the predicted two-dimensional projections and three-dimensional points of the three-dimensional bounding box that correspond to the predicted two-dimensional projections.

15 . The method of claim 14 , further comprising determining the three-dimensional points of the three-dimensional bounding box by predicting one or more spatial dimensions of the three-dimensional bounding box and using the one or more spatial dimensions to scale a unit cube.

16 . An apparatus for selecting a three-dimensional model, comprising:
a memory configured to store an image including an object; and a processor configured to:
obtain the input image including the object;
determine a pose of the object in the input image;
obtain one or more candidate three-dimensional models representing one or more objects in the determined pose; and
determine, from the one or more candidate three-dimensional models, a candidate three-dimensional model to represent the object in the input image.

17 . The apparatus of claim 16 , further comprising generating an output image based on the candidate three-dimensional model and the input image.

18 . The apparatus of claim 16 , further comprising:
receiving a user input to manipulate the candidate three-dimensional model; and adjusting one or more of a pose or a location of the candidate three-dimensional model in an output image based on the user input.

19 . The apparatus of claim 16 , further comprising:
obtaining an additional input image, the additional input image including the object in one or more of a different pose or a different location than a pose or location of the object in the input image; and adjusting one or more of a pose or a location of the candidate three-dimensional model in an output image based on a difference between the pose or location of the object in the additional input image and the pose or location of the object in the input image.

20 . The apparatus of claim 16 , wherein obtaining the one or more three-dimensional models representing the one or more objects includes:
obtaining a plurality of three-dimensional models representing a plurality of objects; and selecting a subset of the plurality of three-dimensional models as the one or more candidate three-dimensional models.

21 . The apparatus of claim 20 , wherein the processor is further configured to:
determine a category of the object in the input image; and determine one or more categories associated with the plurality of candidate three-dimensional models; wherein the one or more candidate three-dimensional models are selected

Assignees

• Qualcomm Technologies Inc

Inventors

• Grabner Alexander
• Roth Peter Michael
• Lepetit Vincent

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V20/653

  by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces · CPC title

• G06T19/20 Primary

  Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• G06T2219/2016

  Rotation, translation, scaling · CPC title