Facial image data generation using partial frame data and landmark data

Patent metadata

Field	Value
Publication number	US-11734952-B1
Application number	US-202217821255-A
Country	US
Kind code	B1
Filing date	Aug 22, 2022
Priority date	Oct 24, 2019
Publication date	Aug 22, 2023
Grant date	Aug 22, 2023

Abstract

Examples described herein include systems for reconstructing facial image data from partial frame data and landmark data. Systems for generating the partial frame data and landmark data are described. Neural networks may be used to reconstruct the facial image data and/or generate the partial frame data. In this manner, compression of facial image data may be achieved in some examples.

First claim

What is claimed is:

1. A method comprising:
receiving difference partial frame data corresponding to a difference between a first video frame and a second video frame, the first video frame including a first facial image associated with a face, the second video frame including a second facial image associated with the face;
receiving difference landmark data corresponding to a difference between second landmark data associated with the second facial image and first landmark data associated with the first facial image, the first landmark data corresponding to facial positions of the first facial image in the first video frame, the facial positions configured to change across one or more frames based on expressions;
utilizing the difference partial frame data and the difference landmark data to reconstruct the second facial image for the second video frame; and
storing the reconstructed second facial image in a memory.

2. The method of claim 1 , wherein utilizing the difference partial frame data and the difference landmark data to reconstruct the second facial image for the second video frame comprises utilizing a trained neural network to reconstruct the second facial image for the second video frame.

3. The method of claim 1 , wherein the first video frame comprises a frame of conference call content.

4. The method of claim 1 , further comprising:
reconstructing the first facial image for the first video frame by combining base facial information and the first landmark data, the base facial information being representative of the first facial image unrelated to expressions in the first video frame; and
storing the reconstructed first facial image in the memory.

5. The method of claim 4 , further comprising:
receiving partial frame data including the base facial information.

6. The method of claim 5 , wherein the partial frame data comprises compressed data associated with the first video frame.

7. The method of claim 5 , wherein the partial frame data comprises the base facial information derived from multiple frames of video content including a face corresponding to the first facial image in the first video frame.

8. The method of claim 5 , wherein the partial frame data comprises a vector representation of the first facial image.

9. A method comprising:
generating difference partial frame data based on changes in partial frame data, the partial frame data including base facial information representative of a facial image from a first video frame including a facial image of a face, the difference partial frame data corresponding to a difference between the first video frame and a second video frame including a second facial image associated with the face;
generating difference landmark data corresponding to a difference between second landmark data associated with the second facial image and first landmark data, the first landmark data corresponding to facial positions of the facial image in the first video frame and across one or more additional frames of video, the facial positions being configured to change across the first video frame and the one or more additional frames based on expressions;
reconstructing one or more representative images of the face corresponding to images of the face in the one or more additional frames of video including combining the base facial information and the first landmark data, the one or more representative images including the facial image;
utilizing the difference partial frame data and the difference landmark data to reconstruct the second facial image; and
storing the reconstructed one or more representative images and the reconstructed second facial image in a memory.

10. The method of claim 9 , further comprising:
transmitting the difference landmark data and the difference partial frame data to a receiving device.

11. The method of claim 10 , wherein the difference landmark data is transmitted at a frame rate matching a video stream frame rate of video data.

12. The method of claim 9 , wherein generating the difference partial frame data comprises tracking changes in the partial frame data.

13. The method of claim 9 , wherein generating the difference landmark data comprises tracking changes in the landmark data.

14. The method of claim 9 , wherein the landmark data corresponds to mapped facial points on the face and changes in the landmark data include coordinate changes for the mapped facial points.

15. The method of claim 9 , wherein the one or more additional frames of video comprise frames of a user in a conference call.

16. The method of claim 9 , further comprising:
generating the partial frame data utilizing a deep neural network configured to recognize the face.

17. A system comprising:
a processor; and
at least one computer readable media encoded with instructions which, when executed, cause the system to perform operations comprising:
providing difference partial frame data corresponding to a difference between at least a first video frame and a second video frame, the at least the first video fame including a first facial image associated with a face, the second video frame including a second facial image associated with the face;
providing difference landmark data corresponding to a difference between first landmark data associated with the at least the first video frame and second landmark data associated with the second video frame, the first landmark data corresponding to facial positions of the first facial image in the at least the first video frame;
generating second video frame data including second image data reconstructed by a trained neural network using the difference partial frame data and the difference landmark data, the second image data including the second facial image for the second video frame; and
displaying the second video frame data including the reconstructed second image data.

18. The system of claim 17 , wherein the first video frame comprises an image of the face against a predetermined background.

19. The system of claim 17 , wherein the operations further comprise:
receiving subsequent difference landmark data associated with the face in subsequent video frames;
reconstructing subsequent images of the face using the subsequent difference landmark data; and
displaying the subsequent images of the face.

20. The system of claim 17 , wherein the difference partial frame data is smaller in size than pixel data associated with the first video frame.

Assignees

• Meta Platforms Tech Llc

Inventors

• Albuz Elif
• Anenberg Bryan Michael
• Guo Peihong
• Kassis Ronit

Classifications

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/094

  Adversarial learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title