Generating animated three-dimensional models from captured images

Patent metadata

Field	Value
Publication number	US-10430642-B2
Application number	US-201815934521-A
Country	US
Kind code	B2
Filing date	Mar 23, 2018
Priority date	Dec 7, 2017
Publication date	Oct 1, 2019
Grant date	Oct 1, 2019

Abstract

A three-dimensional model (e.g., motion capture model) of a user is generated from captured images or captured video of the user. A machine learning network may track poses and expressions of the user to generate and refine the three-dimensional model. Refinement of the three-dimensional model may provide more accurate tracking of the user's face. Refining of the three-dimensional model may include refining the determinations of poses and expressions at defined locations (e.g., eye corners and/or nose) in the three-dimensional model. The refining may occur in an iterative process. Tracking of the three-dimensional model over time (e.g., during video capture) may be used to generate an animated three-dimensional model (e.g., an animated puppet) of the user that simulates the user's poses and expressions.

First claim

What is claimed is:

1. A method, comprising:
obtaining at least one image of a face of a user using a camera located on a device, the device comprising a computer processor, a memory, and a display;
encoding, using the computer processor, the at least one image to generate one or more first feature vectors, wherein the first feature vectors represent one or more facial features of the user in the at least one image;
determining, using the computer processor, a pose of the face of the user and one or more muscle activations of the face of the user in the at least one image from the first feature vectors;
generating, using the computer processor, a three-dimensional model of the user's face based on the determined pose and muscle activations for the user's face;
projecting, using the computer processor, the three-dimensional model onto the at least one image;
defining, using the computer processor, one or more selected locations on the three-dimensional model;
using the three-dimensional model projected onto the at least one image, encoding, at least once, using the computer processor, the at least one image at the selected locations to generate one or more second feature vectors for the at least one image, wherein the second feature vectors represent one or more facial features of the user at the selected locations in the at least one image;
refining, at least once, using the computer processor, the determination of the pose of the face of the user and the one or more muscle activations of the face of the user in the at least one image using the second feature vectors; and
refining, at least once, using the computer processor, the three-dimensional model of the user's face generated from the at least one image based on the refined pose and muscle activations for the user's face.

2. The method of claim 1 , wherein generating the three-dimensional model of the user's face comprises:
assessing, using the computer processor, a registration loss in the at least one image;
determining, using the computer processor, one or more identity parameters for the user's face in the at least one image, wherein the identity parameters minimize the assessed registration loss; and
generating, using the computer processor, the three-dimensional model of the user's face based on the determined pose and muscle activations for the user's face in combination with the determined identity parameters.

3. The method of claim 2 , wherein assessing the registration loss in the at least one image comprises assessing registration loss between the at least one image and at least one additional three-dimensional image of the face of the user.

4. The method of claim 2 , wherein determining the identity parameters comprises backpropagating the registration loss into the three-dimensional model to refine the identity parameters.

5. The method of claim 2 , further comprising refining the determination of the pose of the face of the user and the one or more muscle activations of the face of the user by backpropagating the registration loss into the three-dimensional model.

6. The method of claim 1 , wherein determining the pose and muscle activations comprises performing regression on the feature vectors.

7. The method of claim 1 , wherein projecting the three-dimensional model onto the at least one image is based on parameters of the camera.

8. The method of claim 1 , wherein (a) comprises refining the determination of the pose of the face of the user and the one or more muscle activations of the face of the user using the second feature vectors and (b) comprises refining the three-dimensional model of the user's face generated from the at least one image based on the refined pose and muscle activations for the user's face, and wherein (a) and (b) are repeated a selected number of times.

9. A device, comprising:
a camera;
a display; and
circuitry coupled to the camera and the display, wherein the circuitry is configured to:
obtain a plurality of images of a face of a user using the camera;
for two or more of the images:
generate one or more first feature vectors, wherein the first feature vectors represent one or more facial features of the user in an image;
determine a pose of the face of the user and one or more muscle activations of the face of the user in the at least one image using the first feature vectors;
generate a three-dimensional model of the user's face based on the determined pose and muscle activations for the user's face;
generate, at least once, one or more second feature vectors for the at least one image at one or more selected locations on the three-dimensional model using a projection of the three-dimensional model onto the at least one image, wherein the second feature vectors represent one or more facial features of the user at the selected locations in the at least one image;
refine, at least once, the determination of the pose of the face of the user and the one or more muscle activations of the face of the user in the at least one image using the second feature vectors;
refine, at least once, the three-dimensional model of the user's face generated from the at least one image based on the refined pose and muscle activations for the user's face;

generate an animated three-dimensional model of the face of the user using the refined three-dimensional models generated for the two or more images; and
display a representation of the animated three-dimensional model on the display.

10. The device of claim 9 , wherein the images comprise images from a video of the user captured by the camera.

11. The device of claim 10 , wherein the representation of the animated three-dimensional model displayed on the display comprises a simulation of motion of the user's face from the video of the user.

12. The device of claim 10 , wherein the representation of the animated three-dimensional model displayed on the display comprises a simulation of poses and facial movements of the user's face from the video of the user.

13. The device of claim 9 , wherein the representation of the animated three-dimensional model displayed on the display comprises an animated puppet generated from the animated three-dimensional model of the user.

14. The device of claim 9 , wherein the selected locations comprise locations of interest in the three-dimensional model.

15. A method, comprising:
obtaining at least one image of a face of a user using a camera located on a device, the device comprising a computer processor, a memory, and a display;
generating, using the computer processor, one or more first feature vectors from the at least one image, wherein the first feature vectors represent one or more facial features of the user in the at least one image;
determining, using the computer processor, a pose of the face of the user, one or more muscle activations of the face of the user, and one or more identity parameters for the user's face from the first feature vectors;
generating, using the computer processor, a three-dimensional model of the user's face based on the determined pose, muscle activations, and identity parameters for the user's face;
generating, at least once, using the computer processor, one or more second feature vectors for the at least one image at one or more selected locations on the user's face in the at least one image, wherein the selected locations correspond to locations defined on the three-dimensional model of the user's face, wherein the second feature vectors represent one or more facial features of the user at the selected locations in the at least one image;
refining, at least once, using the computer

Assignees

• Apple Inc

Inventors

• Bouaziz Sofien
• Amberg Brian
• Weise Thibaut
• Snape Patrick
• Brugger Stefan
• Mansfield Alex
• Knothe Reinhard
• Kiser Thomas

Classifications

• G06T17/20 Primary

  Finite element generation, e.g. wire-frame surface description, {tesselation} · CPC title

• G06K9/00281 Primary

  Physics · mapped topic

• G06K9/00261

  Physics · mapped topic

• G06K9/00315

  Physics · mapped topic

• G06V40/171 Primary

  Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships · CPC title