Projecting augmentation images onto moving objects

Patent metadata

Field	Value
Publication number	US-10380802-B2
Application number	US-201815906663-A
Country	US
Kind code	B2
Filing date	Feb 27, 2018
Priority date	Oct 11, 2016
Publication date	Aug 13, 2019
Grant date	Aug 13, 2019

Abstract

A method of augmenting a human face with projected light is disclosed. The method includes determining a blend of component attributes to define visual characteristics of the human face, modifying an input image based, at least in part, on an image of the human face, wherein the modified input image defines an augmented visual characteristic of the human face, determining a present location of one or more landmarks on the human face based, at least in part, on the image of the human face, predicting a future location of the one or more landmarks, deforming a model of the human face based on the future location of the one or more landmarks, generating an augmentation image based on the deformed model and the modified input image, and transmitting for projection the augmentation image.

First claim

What is claimed is:

1. A method to project images onto a human face comprising:
capturing by a camera a first orientation image of infrared light reflecting off of at least a portion of the human face in an initial position;
detecting by a processing element a plurality of landmarks of the human face in the initial position in the first orientation image;
predicting by the processing element a future location of the plurality of landmarks of the human face at a future instance in time;
deforming by the processing element a mesh corresponding to the human face based on the predicted future location of the plurality of landmarks;
generating by the processing element an output image based on the deformed mesh and an input image; and
projecting by a projector in communication with the processing element the output image onto the human face in a future position.

2. The method of claim 1 , further comprising:
determining by the processing element a blend of component attributes defining visual characteristics of the human face in the initial position; and
modifying by the processing element an input image based on the blend of component attributes to generate a modified input image.

3. The method of claim 2 , wherein generating the output image comprises overlaying the modified input image on the deformed mesh.

4. The method of claim 2 , wherein at least two of: determining the blend of component attributes, modifying the input image based on the blend of component attributes, predicting the future location of the plurality of landmarks, deforming the mesh based on the future location of the plurality of landmarks, or overlaying the modified input image on the deformed mesh are performed simultaneously.

5. The method of claim 1 , wherein the camera and the projector are optically aligned with one another.

6. The method of claim 1 , further comprising:
filtering visible light reflecting from the target corresponding to the output image; and
capturing by the camera a second orientation image of the infrared light reflecting off of at least a portion of the human face in a new initial position.

7. The method of claim 1 , wherein predicting the future location of the plurality of landmarks comprises executing a Kalman filter.

8. A projection system for augmenting an appearance of a moving actor comprising:
a lighting source that emits non-visible wavelengths of light onto the moving actor;
a camera that captures an orientation image corresponding to reflection of the non-visible wavelengths off of the moving actor;
a computer in electrical communication with the camera, wherein the computer performs the following operations:
determine a blend of component attributes to define visual characteristics of the moving actor;
modify an input image based on the orientation image to generate a modified input image;
determine a present location of a plurality of facial landmarks on the actor based on the orientation image;
predict a future location of the plurality of facial landmarks;
deform a mesh model corresponding to the actor based on the predicted future location of the landmarks;
overlay the modified input image over the deformed mesh model to generate an output image; and

a projector in electrical communication with the computer, wherein the projector emits visible wavelengths of light corresponding to the output image onto the moving actor.

9. The projection system of claim 8 , wherein the computer executes two or more of the operations in parallel.

10. The projection system of claim 8 , wherein the computer executes a Kalman filter to predict the future location of the plurality of facial landmarks.

11. The projection system of claim 8 , wherein the camera and the projector are optically aligned by a beam splitter.

12. The projection system of claim 8 , wherein the computer further initially optimizes the orientation image.

13. The projection system of claim 8 , wherein the computer further determines a reduced set of component attributes, wherein visual characteristics of the one or more points of the input image are adjusted based on the reduced set of component attributes.

14. The projection system of claim 8 , wherein predicting a future location of the plurality of facial landmarks comprises:
determining an estimated position, velocity, and acceleration for each facial landmark; and
computing a future location at a future time based on the estimated position, velocity, and acceleration.

15. A visual augmentation system comprising:
a camera;
a projector; and
a processor electrically coupled to the camera and the projector, the processor configured to:
receive, from the camera, an image of a human face based on reflected infrared light;
determine a weighted blend of component attributes to approximate the human face based on the received image;
modify a texture based on the weighted blend of component attributes;
detect a location of at least one facial feature the image of the human face;
predict a future location of the at least one facial feature;
generate a deformed model of the human face based on the future location of the at least one facial feature;
apply the texture to the deformed model to generate a projectable image; and
transmit for projection by the projector, the projectable image.

16. The visual augmentation system of claim 15 , further comprising:
a filter positioned in an optical path between the human face and the camera.

17. The visual augmentation system of claim 15 , further comprising:
a plurality of illumination sources to illuminate the human face with infrared light detectable by the camera.

Assignees

• Disney Entpr Inc

Inventors

• Grundhöfer Anselm
• Bermano Amit

Classifications

• G06T7/33

  using feature-based methods · CPC title

• H04N5/33

  Transforming infrared radiation (cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20) · CPC title

• G06T7/50

  Depth or shape recovery · CPC title

• G06T11/00

  Two-dimensional [2D] image generation · CPC title

• G06T2207/30201

  Face · CPC title