Pose synthesis

Patent metadata

Field	Value
Publication number	US-11030772-B2
Application number	US-201916429696-A
Country	US
Kind code	B2
Filing date	Jun 3, 2019
Priority date	Jun 3, 2019
Publication date	Jun 8, 2021
Grant date	Jun 8, 2021

Abstract

Examples are disclosed that relate to computing devices and methods for synthesizing a novel pose of an object. One example provides a method comprising receiving a reference image of an object corresponding to an original viewpoint. The reference image of the object is translated into a depth map of the object, and a new depth map of the object is synthesized to correspond to a new viewpoint. A new image of the object is generated from the new viewpoint based on the new depth map of the object and the reference image of the object.

First claim

The invention claimed is:

1. Enacted on a computing system, a method for synthesizing a novel pose of an object, the method comprising:
receiving a reference image of an object corresponding to an original viewpoint;
translating the reference image of the object into a reference depth map of the object;
synthesizing a new depth map of the object corresponding to a new viewpoint; and
inputting the reference image of the object and the new depth map of the object into an identity recovery model to generate a new image of the object from the new viewpoint.

2. The method of claim 1 , wherein translating the reference image of the object into the reference depth map of the object comprises:
inputting the reference image of the object into a domain transfer module; and
receiving the reference depth map of the object from the domain transfer module.

3. The method of claim 2 , further comprising receiving a foreground mask from the domain transfer module, the foreground mask identifying pixels associated with the object.

4. The method of claim 2 , wherein the domain transfer module comprises a domain transfer model, the method further comprising training the domain transfer model on a dataset of paired images and depth maps.

5. The method of claim 1 , wherein synthesizing the new depth map of the object corresponding to the new viewpoint comprises:
inputting the reference depth map of the object into a depth map generator;
receiving the new depth map of the object from the depth map generator; and
refining the new depth map of the object using a 3D depth refinement module.

6. The method of claim 5 , further comprising receiving a sequence of new depth maps from the depth map generator, and wherein refining the new depth map comprises using a 3D convolutional neural network to enforce consistency among the sequence of new depth maps.

7. The method of claim 1 , wherein generating the new image of the object comprises:
mapping the reference image of the object to an appearance parameter;
mapping the new depth map of the object to a shape parameter; and
combining the shape parameter and the appearance parameter to generate the new image of the object from the new viewpoint.

8. The method of claim 1 , further comprising training the identity recovery model on unpaired depth and image data.

9. The method of claim 1 , further comprising training the identity recovery model by:
using a first structure encoder to map the reference image to a reference shape parameter;
using a second structure encoder to map the new depth map to a new shape parameter;
using a first appearance encoder to map the reference image to a reference appearance parameter;
using a second appearance encoder to map the new depth map to a new appearance parameter; and
combining each of the reference shape parameter and the new shape parameter with one of the reference appearance parameter and the new appearance parameter to generate an image.

10. The method of claim 1 , further comprising training the identity recovery model using supervised learning and unsupervised learning.

11. The method of claim 10 , further comprising directly supervising the training using the reference image of the object, the reference depth map, and the new depth map.

12. A computing device comprising:
a processor; and
a storage device storing instructions executable by the processor to
receive a reference image of an object corresponding to an original viewpoint;
translate the reference image of the object into a reference depth map of the object;
synthesize a new depth map of the object corresponding to a new viewpoint; and
input the reference image of the object and the new depth map of the object into an identity recovery model to generate a new image of the object from the new viewpoint.

13. The computing device of claim 12 , wherein the instructions are further executable to translate the reference image of the object into the reference depth map of the object by:
inputting the reference image of the object into a domain transfer module; and
receiving the reference depth map of the object from the domain transfer module.

14. The computing device of claim 13 , wherein the instructions are further executable to receive a foreground mask from the domain transfer module, the foreground mask identifying pixels associated with the object.

15. The computing device of claim 12 , wherein the instructions are further executable to generate the new image of the object by:
mapping the reference image of the object to an appearance parameter;
mapping the new depth map of the object to a shape parameter; and
combining the shape parameter and the appearance parameter to generate the new image of the object from the new viewpoint.

16. The computing device of claim 12 , wherein the instructions are further executable to train the identity recovery model by:
using a first structure encoder to map the reference image to a reference shape parameter;
using a second structure encoder to map the new depth map to a new shape parameter;
using a first appearance encoder to map the reference image to a reference appearance parameter;
using a second appearance encoder to map the new depth map to a new appearance parameter; and
combining each of the reference shape parameter and the new shape parameter with one of the reference appearance parameter and the new appearance parameter to generate an image.

17. The computing device of claim 16 , wherein the instructions are further executable to directly supervise the training using the reference image of the object, the reference depth map, and the new depth map.

18. A computing device comprising:
a processor; and
a storage device storing instructions executable by the processor to
receive a reference image of an object corresponding to an original viewpoint;
translate the reference image of the object into a reference depth map of the object;
synthesize a new depth map of the object corresponding to a new viewpoint; and
input the reference image of the object and the new depth map of the object into an identity recovery model to generate a new image of the object from the new viewpoint, wherein the instructions are further executable to generate the new image of the object by:
mapping the reference image of the object to an appearance parameter;
mapping the new depth map of the object to a shape parameter; and
combining the shape parameter and the appearance parameter to generate a new image of the object from the new viewpoint.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Dixit Mandar Dilip
• Liu Bo
• Hua Gang

Classifications

• G06T2207/30244

  Camera pose · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06T2200/04

  involving 3D image data · CPC title

• G06T17/00 Primary

  Three-dimensional [3D] modelling for computer graphics · CPC title

• G06T7/73

  using feature-based methods · CPC title