Three dimensional modelling

Patent metadata

Field	Value
Publication number	US-10521649-B2
Application number	US-201615551247-A
Country	US
Kind code	B2
Filing date	Feb 16, 2016
Priority date	Feb 16, 2015
Publication date	Dec 31, 2019
Grant date	Dec 31, 2019

Abstract

Two related methods of fitting a three dimensional model, and a method of performing facial recognition, are disclosed. One method comprises estimating and refining geometric information using image landmarks on an object in a two dimensional image. The other method comprises estimating and refining photometric information of the object in the two dimensional image. Furthermore, a method of performing image recognition is provided.

First claim

The invention claimed is:

1. A method of fitting a three dimensional 3D model to an object in a two dimensional 2D image, the method comprising:
detecting a plurality of image landmarks on the object in the 2D image;
estimating 3D geometric information of the plurality of image landmarks from the 2D image and known geometric information of a base model; and
refining the geometric information of the base model in dependence on a cost function relating to a distance in 3D space between the image landmarks and corresponding model landmarks on the base model, based on the estimated 3D geometric information of the plurality of image landmarks, so as to reduce the distance in 3D space between the image landmarks and the corresponding model landmarks.

2. The method of claim 1 , wherein estimating 3D geometric information comprises:
projecting the plurality of image landmarks onto the surface of the generic 3D model in order to estimate a depth of said image landmark in three-dimensional space.

3. The method of claim 1 , wherein refining the geometric information of the base model comprises minimising the distance between the image landmarks and the corresponding model landmarks by iteratively adjusting the pose and shape of the base model.

4. The method of claim 1 , wherein the plurality of image landmarks are on the contour edge of the object in the 2D image.

5. The method of claim 1 , wherein estimating 3D geometric information of a plurality of image landmarks comprises estimating 3D geometric information of between 2 and 100 image landmarks.

6. The method of claim 1 , further comprising:
estimating light direction and light strength of the object from the 2D image;
estimating the albedo of the object based on the estimated light direction and light strength;
iteratively refining the estimation of light direction and light strength using the estimated albedo; and
applying the refined light direction, light strength and albedo to the base model.

7. The method of claim 6 , wherein estimating the light direction and light strength comprises linearising a Phong model by keeping specular reflectance and shininess constant across the iterations.

8. The method of claim 1 , wherein the object is a human face.

9. A method of performing facial recognition, the method comprising:
generating a 3D model according to claim 8 ;
rotating the 2D image to a frontal view using the 3D model;
extracting local features from the 2D image; and
comparing the local features with images in a database.

10. A non-transitory computer readable storage medium arranged to store computer program instructions which, when executed on one or more processors, perform the method according to claim 1 .

11. Apparatus for fitting a three dimensional 3D model to an object in a two dimensional 2D image, the apparatus comprising:
one or more processors; and
computer-readable memory arranged to store computer program instructions which, when executed by the one or more processors, cause the apparatus to:
detect a plurality of image landmarks on the object in the 2D image;
estimate 3D geometric information of the plurality of image landmarks from the 2D image and known geometric information of a base model; and
refine the geometric information of the base model in dependence on a cost function relating to a distance in 3D space between the image landmarks and corresponding model landmarks on the base model, based on the estimated 3D geometric information of the plurality of image landmarks, so as to reduce the distance in 3D space between the image landmarks and the corresponding model landmarks.

12. The apparatus of claim 11 , wherein estimating 3D geometric information comprises:
projecting the plurality of image landmarks onto the surface of the generic 3D model in order to estimate a depth of said image landmark in three-dimensional space.

13. The apparatus of claim 11 , wherein refining the geometric information of the base model comprises minimising the distance between the image landmarks and the corresponding model landmarks by iteratively adjusting the pose and shape of the base model.

14. The apparatus of claim 11 , wherein the plurality of image landmarks are on the contour edge of the object in the 2D image.

Assignees

• Univ Surrey

Inventors

• Kittler Josef
• Christmas William Jeffrey
• Yan Fei
• Hu Guosheng

Classifications

• G06V40/172

  Classification, e.g. identification · CPC title

• G06T7/33 Primary

  using feature-based methods · CPC title

• G06T2207/10028

  Range image; Depth image; 3D point clouds · CPC title

• G06T7/514

  from specularities · CPC title

• G06T2207/30201

  Face · CPC title