Method and apparatus for adjusting face pose

Patent metadata

Field	Value
Publication number	US-10134177-B2
Application number	US-201614996709-A
Country	US
Kind code	B2
Filing date	Jan 15, 2016
Priority date	Jan 15, 2015
Publication date	Nov 20, 2018
Grant date	Nov 20, 2018

Abstract

A method and an apparatus for adjusting a pose in a face image are provided. The method of adjusting a pose in a face image involves detecting two-dimensional (2D) landmarks from a 2D face image, positioning three-dimensional (3D) landmarks in a 3D face model by determining an initial pose of the 3D face model based on the 2D landmarks, updating the 3D landmarks by iteratively adjusting a pose and a shape of the 3D face model, and adjusting a pose in the 2D face image based on the updated 3D landmarks.

First claim

What is claimed is:

1. A method of adjusting a pose in a face image, comprising:
measuring first deviations between two-dimensional (2D) landmarks from a 2D face image of a face and corresponding 2D landmarks from a first sample 2D face image
positioning three-dimensional (3D) landmarks in a 3D face model and adjusting a pose of the 3D face model by determining initial pose information of the 3D face model based on the first deviations;
acquiring a 2D texture image by two-dimensionally projecting the 3D face model with the adjusted pose;
measuring second deviations between new 2D landmarks from the 2D texture image and corresponding 2D landmarks from a second sample 2D face image;
updating positions of the 3D landmarks and adjusting the pose of the 3D face model by determining updated pose information based on the second deviations; and
adjusting a pose in the 2D face image based on the 3D landmarks.

2. The method of claim 1 , wherein the positioning of the 3D landmarks comprises determining an initial geometric parameter representing an initial pose of the 3D face model based on the 2D landmarks.

3. The method of claim 2 , wherein the initial geometric parameter comprises the initial pose information on the 3D face model, and the initial pose information is represented by a pose information vector comprising roll information, pitch information, and yaw information on the 3D face model.

4. The method of claim 2 , wherein the determining of the initial geometric parameter comprises:
determining an initial weight for the 2D landmarks based on the 2D landmarks; and
determining the initial geometric parameter that minimizes a first energy function based on the initial weight, the 2D landmarks, and the 3D landmarks.

5. A method of adjusting a pose in a face image, comprising:
detecting two-dimensional (2D) landmarks from a 2D face image;
positioning three-dimensional (3D) landmarks in a 3D face model and adjusting a pose of the 3D face model by determining an initial pose of the 3D face model based on the 2D landmarks;
updating the 3D landmarks by iteratively adjusting a pose and a shape of the 3D face model; and
adjusting a pose in the 2D face image based on the updated 3D landmarks,
wherein the positioning of the 3D landmarks comprises determining an initial geometric parameter representing the initial pose of the 3D face model based on the 2D landmarks,
wherein the determining of the initial geometric parameter comprises:
determining an initial weight for the 2D landmarks based on the 2D landmarks, and
determining the initial geometric parameter that minimizes a first energy function based on the initial weight, the 2D landmarks, and the 3D landmarks, and

wherein the determining of the initial weight comprises:
measuring deviations between the 2D landmarks and an actually detected landmark of a 2D face sample image stored in a face image database; and
determining the initial weight for the 2D landmarks based on the measured deviations, and

wherein the updating of the 3D landmarks comprises:
acquiring a 2D texture image by two-dimensionally projecting the 3D face model with the adjusted pose;
measuring second deviations between new 2D landmarks from the 2D texture image and corresponding 2D landmarks from a second sample 2D face image.

6. The method of claim 5 , wherein the determining of the initial weight for the 2D landmarks based on the measured deviations comprises:
arranging the 2D landmarks based on the measured deviation;
selecting some 2D landmarks having measured deviations smaller than a preset threshold among the arranged 2D landmarks; and
assigning the initial weight to the selected 2D landmarks in inverse proportion to measured deviations.

7. The method of claim 5 , wherein the face image database comprises at least one of a plurality of 2D face sample images, actually detected landmarks from the 2D face sample images, and actually measured geometric parameters from the 2D face sample images.

8. The method of claim 2 , wherein the updating of the 3D landmarks comprises:
updating the 3D landmarks based on the initial geometric parameter;
determining a weight for the updated 3D landmarks; and
determining, based on the weight, a geometric parameter and a shape parameter of the 3D face model that minimize a second energy function.

9. The method of claim 8 , wherein the updating of the 3D landmarks based on the initial geometric parameter comprises:
adjusting the pose of the 3D face model based on the initial geometric parameter;
the acquiring of the 2D texture image by two-dimensionally projecting the 3D face model with the adjusted pose;
detecting the new 2D landmarks from the 2D texture image;
retrieving 3D landmarks corresponding to the new 2D landmarks; and
updating the 3D landmarks using the retrieved 3D landmarks.

10. The method of claim 9 , wherein the determining of the weight for the updated 3D landmarks comprises:
acquiring current pose information on the 3D face model from the initial geometric parameter;
retrieving pose information substantially corresponding to the current pose information on the 3D face model from a previously stored face image database;
retrieving the second sample 2D face image corresponding to the retrieved pose information from the face image database; and
determining a weight for 2D landmarks corresponding to the retrieved sample 2D face image based on the second deviations between the 2D landmarks and the new 2D landmarks.

11. The method of claim 10 , wherein the determining of the weight for the 2D landmarks comprises:
calculating an average of the second deviations; and
determining the weight for the 2D landmarks in inverse proportion to the average of the second deviations.

12. The method of claim 10 , wherein the pose information substantially corresponding to the current pose information comprises pose information with a deviation from the current pose information being within a preset threshold range.

13. The method of claim 8 , wherein the updating of the 3D landmarks comprises iteratively adjusting the pose and shape of the 3D face model based on whether a deviation between a geometric parameter of the 3D face model acquired in (i-2)th iteration and a geometric parameter acquired in (i-1)th iteration is smaller than a preset threshold or whether iteration is performed a preset number of times.

14. The method of claim 13 , wherein the adjusting of the pose in the 2D face image comprises:
reconstructing the 3D face model based on a final geometric parameter and a final shape parameter calculated in final iteration; and
adjusting the pose in the 2D face image by two-dimensionally projecting the reconstructed 3D face model.

15. The method of claim 14 , further comprising mapping a background preserving texture to the 2D face image with the adjusted pose.

16. The method of claim 15 , wherein the mapping of the background preserving texture comprises determining a 3D texture model by imparting a color to the reconstructed 3D face model based on a pixel color of the 2D face image; and determining a pixel color for a background region other than a face region in a face image obtained by twodimensionally projecting the 3D texture model.

17. The method of claim 16 , wherein the determining of the 3D texture model comprises:
fitting vertexes of the reconstructed 3D face model to the 2D face image based on the final geometric parameter and the final shape parameter;
assigning a texture of the fitted 2D face image to the vertexes of the reconstructed 3D face model; determi

Assignees

• Samsung Electronics Co Ltd

Inventors

• Feng Xuetao
• Feng Hao
• Zhang Hongwei
• Gong Lujin
• Choi Changkyu
• Han Jae-Joon
• Suh Sungjoo
• Hwang Wonjun
• Wang Biao

Classifications

• G06V10/54 Primary

  relating to texture · CPC title

• G06T15/10 Primary

  Geometric effects · CPC title

• G06V10/7553

  based on shape, e.g. active shape models [ASM] · CPC title

• G06V10/7557

  based on appearance, e.g. active appearance models [AAM] · CPC title

• G06K9/00208

  Physics · mapped topic