System and method for processing video to provide facial de-identification

Patent metadata

Field	Value
Publication number	US-9799096-B1
Application number	US-201514794165-A
Country	US
Kind code	B1
Filing date	Jul 8, 2015
Priority date	Jul 8, 2014
Publication date	Oct 24, 2017
Grant date	Oct 24, 2017

Abstract

A system and method for real-time image and video face de-identification that removes the identity of the subject while preserving the facial behavior is described The facial features of the source face are replaced with that of the target face while preserving the facial actions of the source face on the target face. The facial actions of the source face are transferred to the target face using personalized Facial Action Transfer (FAT), and the color and illumination is adapted. Finally, the source image or video containing the target facial features is outputted for display. Alternatively, the system can run in real-time.

First claim

The invention claimed is:

1. A method for face facial de-identification in an image or video comprising:
obtaining an image of a source face represented as a plurality of source pixels;
obtaining an image of a target face represented as a plurality of target pixels;
computing the shape deformations of the source face with respect to a neutral face using triangle-based deformation and applying the same triangle-based deformation to the neutral face to deform the target face; and
adapting illumination and color characteristics of the source face to the target face by:
establishing a plurality of 2D landmarks on the source face and target face;
for each of the source face and target face, estimating a plurality of 3D landmarks corresponding to the 2D landmarks and representing a surface over the 3D landmarks as a spherical surface;
aligning source pixels to target pixels using triangle warping and calculating the corresponding pixels between the source pixels and target pixels;
projecting the source pixels and target pixels onto their respective spherical surfaces and computing, for each source and target pixel, a harmonic basis;
calculating a plurality of harmonic coefficients representing an approximation of the intensity of source pixels in each of the RGB color channels; and
constructing the pixel intensity for the target pixels as the product of the harmonic basis for the target pixels and the harmonic coefficients calculated with respect to the source pixels.

2. The method of claim 1 wherein the step of computing the shape deformations comprises:
computing a mapping between a plurality of triangles modelling said source face between a neutral expression and another facial expression; and
applying said mapping to said target face to produce a substantially similar facial expression on said target face.

3. The method of claim 2 wherein said mapping is generated by learning a regression between said neutral expression and said other facial expressions.

4. The method of claim 1 further comprising:
encoding a password in a de-identified image; and
permitting access to an identified image showing said source face when said password is input by a user.

5. The method of claim 1 comprising a prior step of performing facial feature tracking on said source face until a change in facial expression is detected.

6. A system for providing personalized publicity to a user comprising:
a display, for displaying one or more advertisements to said user;
a processor, running a program for providing the following functions:
obtaining an image of said user;
replacing the face of a person appearing in said advertisements with said image of said user; and
transferring facial expressions of said person in said advertisements to said image of said user;
wherein the functions of replacing and transferring use the method of claim 1 .

7. A system for de-identifying a face appearing in a video stream comprising:
a source of a stream of video;
a video processor running a program which provides the functions of:
detecting said face in said stream of video;
replacing said face with an image of an anonymous face; and
transferring facial expressions of said face to said anonymous face;
wherein the functions of replacing and transferring use the method of claim 1 .

8. A system for de-identifying faces in surveillance data comprising:
one or more cameras providing a stream of surveillance data; and
a processor running a program which provides the functions of:
detecting a face in said stream of surveillance data;
replacing said face with an image of an anonymous face; and
transferring facial expressions of said face to said anonymous face;
wherein the functions of replacing and transferring use the method of claim 1 .

9. A system for enhancing a video game comprising:
a video game, rendered locally or online;
a processor running in conjunction with said video game, said processor running a program which provides the functions of:
obtaining an image of a user of said video game;
replacing the face of a person appearing in said video game with said image of said user; and
transferring facial expressions of said person in said video game to said image of said user;
wherein the functions of replacing and transferring use the method of claim 1 .

10. A system for providing communication to users for purposes of long distance learning comprising:
a camera, for obtaining a stream of video containing the face a user;
a video processor running a program which provides the functions of:
detecting the face of said user in said stream of video data;
replacing said face with an image of an anonymous face; and
transferring the facial expressions of said face to said anonymous face; and
a display, for displaying said stream of video data
wherein the functions of replacing and transferring use the method of claim 1 .

11. An apparatus comprising:
a processor; and
memory containing logic for execution by the processor, the logic configured to:
obtain an image of a source face represented as a plurality of source pixels;
obtain an image of a target face represented as a plurality of target pixels;
compute the shape deformations of the source face with respect to a neutral face using triangle-based deformation and apply the same triangle-based deformation to the neutral face to deform the target face; and
adapt illumination and color characteristics of the source face to the target face by:
establishing a plurality of 2D landmarks on the source face and target face;
for each of the source face and target face, estimating a plurality of 3D landmarks corresponding to the 2D landmarks and representing a surface over the 3D landmarks as a spherical surface;
aligning source pixels to target pixels using triangle warping and calculating the corresponding pixels between the source pixels and target pixels;
projecting the source pixels and target pixels onto their respective spherical surfaces and computing, for each source and target pixel, a harmonic basis;
calculating a plurality of harmonic coefficients representing an approximation of the intensity of source pixels in each of the RGB color channels; and
constructing the pixel intensity for the target pixels as the product of the harmonic basis for the target pixels and the harmonic coefficients calculated with respect to the source pixels.

12. The apparatus of claim 11 wherein computing the shape deformations comprises:
computing a mapping between a plurality of triangles modelling said source face between a neutral expression and another facial expression; and
applying said mapping to said target face to produce a substantially similar facial expression on said target face;
wherein said mapping is generated by learning a regression between said neutral expression and said other facial expressions.

13. The apparatus of claim 11 wherein the logic is further configured to:
encode a password in a de-identified image; and
permit access to an identified image showing said source face when said password is input by a user.

Assignees

• Univ Carnegie Mellon

Inventors

• De La Torre Fernando
• Cohn Jeffrey F
• Huang Dong

Classifications

• A63F13/25

  Output arrangements for video game devices · CPC title

• A63F13/655 Primary

  by importing photos, e.g. of the player · CPC title

• G06T2207/20036

  Morphological image processing · CPC title

• G06T2210/44

  Morphing · CPC title

• A63F2300/5553

  user representation in the game field, e.g. avatar · CPC title