Dense reconstruction for narrow baseline motion observations

Patent metadata

Field	Value
Publication number	US-10783704-B2
Application number	US-201816144505-A
Country	US
Kind code	B2
Filing date	Sep 27, 2018
Priority date	Sep 27, 2018
Publication date	Sep 22, 2020
Grant date	Sep 22, 2020

Abstract

Techniques for constructing a three-dimensional model of facial geometry are disclosed. A first three-dimensional model of an object is generated, based on a plurality of captured images of the object. A projected three-dimensional model of the object is determined, based on a plurality of identified blendshapes relating to the object. A second three-dimensional model of the object is generated, based on the first three-dimensional model of the object and the projected three dimensional model of the object.

First claim

What is claimed is:

1. A method, comprising:
generating, using one or more computer processors, a first three-dimensional model of an object based on a plurality of captured images of the object;
determining, using the one or more computer processors, a projected three-dimensional model of the object based on a plurality of identified blendshapes relating to the object;
identifying one or more unreliable values, relating to at least one of color or depth, in the first three-dimensional model by comparing one or more first values, relating to at least one of color or depth, in the first three-dimensional model with one or more second values, relating to at least one of color or depth, in the projected three-dimensional model; and
generating, using the one or more computer processors, a second three-dimensional model of the object correcting the first three-dimensional model, by excluding the one or more unreliable values from the first three-dimensional model.

2. The method of claim 1 , wherein generating the first three-dimensional model of the object further comprises:
receiving a plurality of narrow baseline motion captured images of the object; and
estimating, using the one or more computer processors, image parameters and depth values related to the first three-dimensional model based on analyzing the plurality of narrow baseline motion captured images of the object.

3. The method of claim 1 , wherein determining the projected three-dimensional model of the object further comprises:
identifying, using the one or more computer processors, the plurality of blendshapes related to the object;
determining, using the one or more computer processors, a weight relating to each of the plurality of identified blendshapes; and
determining, using the one or more computer processors, the projected three-dimensional model of the object based on the plurality of identified blendshapes and the determined weights.

4. The method of claim 1 , wherein generating, using the one or more computer processors, the second three-dimensional model of the object correcting the first three-dimensional model further comprises:
identifying one or more unreliable captured images in the plurality of captured images of the object; and
re-generating the first three-dimensional model based on the plurality of captured images, excluding the identified one or more unreliable captured images.

5. The method of claim 1 , wherein the unreliable values, the first values, and the second values each relate to color.

6. The method of claim 1 , wherein the unreliable values, the first values, and the second values each relate to depth.

7. The method of claim 1 , wherein generating the second three-dimensional model of the object is further based on analyzing the plurality of captured images of the object using a trained machine learning model.

8. The method of claim 7 , wherein the trained machine learning model comprises a convolutional neural network.

9. The method of claim 1 , wherein generating the second three-dimensional model of the object is further based on analyzing the plurality of identified blendshapes using a trained machine learning model.

10. A computer program product comprising:
a non-transitory computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform an operation, the operation comprising:
generating a first three-dimensional model of an object based on a plurality of captured images of the object;
determining a projected three-dimensional model of the object based on a plurality of identified blendshapes relating to the object;
identifying one or more unreliable values, relating to at least one of color or depth, in the first three-dimensional model by comparing one or more first values, relating to at least one of color or depth, in the first three-dimensional model with one or more second values, relating to at least one of color or depth, in the projected three-dimensional model; and
generating a second three-dimensional model of the object, correcting the first three-dimensional model, by excluding the one or more unreliable values from the first three-dimensional model.

11. The computer program product of claim 10 , wherein generating the first three-dimensional model of the object further comprises:
receiving a plurality of narrow baseline motion captured images of the object; and
estimating, using the one or more computer processors, image parameters and depth values related to the first three-dimensional model based on analyzing the plurality of narrow baseline motion captured images of the object.

12. The computer program product of claim 10 , wherein determining the projected three-dimensional model of the object further comprises:
identifying, using the one or more computer processors, the plurality of blendshapes related to the object;
determining, using the one or more computer processors, a weight relating to each of the plurality of identified blendshapes; and
determining, using the one or more computer processors, the projected three-dimensional model of the object based on the plurality of identified blendshapes and the determined weights.

13. The computer program product of claim 10 , wherein generating the second three-dimensional model of the object, correcting the first three-dimensional model, further comprises:
identifying one or more unreliable captured images in the plurality of captured images of the object; and
re-generating the first three-dimensional model based on the plurality of captured images, excluding the identified one or more unreliable captured images.

14. The computer program product of claim 10 , wherein the unreliable values, the first values, and the second values each relate to color, and wherein generating the second three-dimensional model of the object, correcting the first three-dimensional model, further comprises:
identifying one or more unreliable depth values relating to the first three-dimensional model by comparing a first one or more depth values relating to the first three-dimensional model with a second one or more depth values relating to the projected three-dimensional model; and
excluding the unreliable depth values from the first three-dimensional model.

15. A system, comprising:
a processor; and
a memory storing a program, which, when executed on the processor, performs an operation, the operation comprising:
generating a first three-dimensional model of an object based on a plurality of captured images of the object;
determining a projected three-dimensional model of the object based on a plurality of identified blendshapes relating to the object;
identifying one or more unreliable values, relating to at least one of color or depth, in the first three-dimensional model by comparing one or more first values, relating to at least one of color or depth, in the first three-dimensional model with one or more second values, relating to at least one of color or depth, in the projected three-dimensional model; and
generating a second three-dimensional model of the object, correcting the first three-dimensional model, by excluding the one or more unreliable values from the first three-dimensional model.

16. The system of claim 15 , wherein generating the first three-dimensional model of the object further comprises:
receiving a plurality of narrow baseline motion captured images of the object; and
estimating, using the processor, image parameters and depth values related to the first three-dimension

Assignees

• Disney Entpr Inc

Inventors

• Mitchell Kenneth J
• Dümbgen Frederike
• Liu Shuang

Classifications

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/08

  Learning methods · CPC title