Face image processing device and face image processing program

Patent metadata

Field	Value
Publication number	US-11682128-B2
Application number	US-202117201487-A
Country	US
Kind code	B2
Filing date	Mar 15, 2021
Priority date	Apr 1, 2020
Publication date	Jun 20, 2023
Grant date	Jun 20, 2023

Abstract

A face image processing device, includes: an image coordinate system coordinate value derivation unit detecting an x-coordinate value and a y-coordinate value in an image coordinate system at a feature point of an organ of a face of a person in an image, and estimating a z-coordinate value, so as to derive three-dimensional coordinate values in the image coordinate system; a camera coordinate system coordinate value derivation unit deriving three-dimensional coordinate values in a camera coordinate system from the three-dimensional coordinate values in the image coordinate system derived by the image coordinate system coordinate value derivation unit; and a parameter derivation unit applying the three-dimensional coordinate values in the camera coordinate system derived by the camera coordinate system coordinate value derivation unit to a predetermined three-dimensional face shape model to derive a model parameter of the three-dimensional face shape model in the camera coordinate system.

First claim

What is claimed is:

1. A face image processing device, comprising:
at least one processor configured to implement:
an image coordinate system coordinate value derivation unit configured to detect an x-coordinate value which is a horizontal coordinate value and a y-coordinate value which is a vertical coordinate value respectively in an image coordinate system at a feature point of an organ of a face of a person in an image acquired by imaging the face, and estimate a z-coordinate value which is a depth coordinate value in the image coordinate system, so as to derive three-dimensional coordinate values in the image coordinate system;
a camera coordinate system coordinate value derivation unit configured to derive three-dimensional coordinate values in a camera coordinate system from the three-dimensional coordinate values in the image coordinate system derived by the image coordinate system coordinate value derivation unit; and
a parameter derivation unit configured to apply the three-dimensional coordinate values in the camera coordinate system derived by the camera coordinate system coordinate value derivation unit to a predetermined three-dimensional face shape model to derive a model parameter of the three-dimensional face shape model in the camera coordinate system.

2. The face image processing device according to claim 1 , wherein
the image coordinate system coordinate value derivation unit is configured to derive the z-coordinate value by estimating the z-coordinate value using deep learning in parallel with the detection of the x-coordinate value and the y-coordinate value.

3. The face image processing device according to claim 1 , wherein
the three-dimensional face shape model is formed of a linear sum of an average shape and a basis, and the basis is separated into an individual difference basis, which is a component that does not change with time, and a facial expression basis, which is a component that changes with time, and
the model parameter includes a parameter of the individual difference basis and a parameter of the facial expression basis.

4. The face image processing device according to claim 3 , wherein
the model parameter further includes a position and orientation parameter indicating a position and an orientation of the face of the person as viewed from a camera that images the face of the person.

5. The face image processing device according to claim 1 , further comprising:
a face distance estimation unit configured to estimate a distance to the face of the person, wherein
the camera coordinate system coordinate value derivation unit is configured to derive the z-coordinate value in the camera coordinate system by using the distance estimated by the face distance estimation unit.

6. The face image processing device according to claim 1 , further comprising:
a coordinate correction unit configured to correct the z-coordinate value, which is derived by the image coordinate system coordinate value derivation unit, according to a position of the face in the image.

7. The face image processing device according to claim 1 , further comprising:
a weight setting unit configured to set independent weights for respective error evaluation values of the x-coordinate value, the y-coordinate value, and the z-coordinate value at each feature point of each organ of the face.

8. A face image processing program stored in a non-transitory computer readable medium configured to cause a computer to execute processing, the processing comprising:
detecting an x-coordinate value which is a horizontal coordinate value and a y-coordinate value which is a vertical coordinate value respectively in an image coordinate system at a feature point of an organ of a face of a person in an image acquired by imaging the face, and estimating a z-coordinate value which is a depth coordinate value in the image coordinate system, so as to derive three-dimensional coordinate values in the image coordinate system;
deriving three-dimensional coordinate values in a camera coordinate system from the derived three-dimensional coordinate values in the image coordinate system; and
applying the derived three-dimensional coordinate values in the camera coordinate system to a predetermined three-dimensional face shape model to derive a model parameter of the three-dimensional face shape model in the camera coordinate system.

Assignees

• Aisin Corp
• Aisin Seiki

Inventors

• Kojima Shin-Ichi
• Yamada Yuya
• Osuga Shin
• Ueda Koki
• Kawai Yuta
• Kato Takashi
• Yamada Yoshiyuki

Classifications

• G06V10/82

  using neural networks · CPC title

• G06T7/70

  Determining position or orientation of objects or cameras (camera calibration G06T7/80) · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06V40/161

  Detection; Localisation; Normalisation · CPC title

• G06V40/171

  Local features and components; Facial parts (eye characteristics G06V40/18); Occluding parts, e.g. glasses; Geometrical relationships · CPC title