Face-based frame rate upsampling for video calls

Patent metadata

Field	Value
Publication number	US-11869274-B2
Application number	US-202217707661-A
Country	US
Kind code	B2
Filing date	Mar 29, 2022
Priority date	Aug 7, 2019
Publication date	Jan 9, 2024
Grant date	Jan 9, 2024

Abstract

A method includes receiving a set of video frames that correspond to a video, including a first video frame and a second video frame that each include a face, wherein the second video frame is subsequent to the first video frame. The method further includes performing face tracking on the first video frame to identify a first face resampling keyframe and performing face tracking on the second video frame to identify a second face resampling keyframe. The method further includes deriving an interpolation amount. The method further includes determining a first interpolated face frame based on the first face resampling keyframe and the interpolation amount. The method further includes determining a second interpolated face frame based on the second face resampling keyframe and the interpolation amount. The method further includes rendering an interpolated first face and an interpolated second face. The method further includes displaying a final frame.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving a set of video frames that correspond to a video, the set of video frames including a first video frame and a second video frame that each include a face, wherein the second video frame is subsequent to the first video frame;
performing face tracking on the first video frame to identify a first face resampling keyframe;
performing face tracking on the second video frame to identify a second face resampling keyframe;
deriving an interpolation amount;
interpolating a first background of the first face resampling keyframe and a second background of the second face resampling keyframe based on the interpolation amount;
rendering an interpolated first face and an interpolated second face; and
displaying a final frame that is based on the interpolated first background, the interpolated second background, the interpolated first face, and the interpolated second face.

2. The method of claim 1 , further comprising:
blending the first background with the second background to obtain a blended background;
blending the interpolated first face with the interpolated second face to obtain a blended interpolated face; and
generating the final frame by placing a smooth face on top of the blended interpolated face and the blended background.

3. The method of claim 1 , wherein the first face resampling keyframe includes a first head transform matrix and first face landmark vertices and further comprising determining a first interpolated face frame by:
using the first head transform matrix to extract a translation vector, a rotation quaternion, and a scale vector;
linearly interpolating the translation vector;
using a linear interpolation to interpolate the rotation quaternion to generate an interpolated rotation quaternion;
linearly interpolating the scale vector to generate an interpolated scale vector;
composing an interpolated translation-rotation-scale matrix based on the interpolated translation vector, the interpolated rotation quaternion, and the interpolated scale vector; and
calculating an interpolated position for the interpolated first face using the interpolated translation-rotation-scale matrix.

4. The method of claim 1 , wherein the interpolation amount is derived from (a) a duration between the first face resampling keyframe and the second face resampling keyframe and (b) a current render time.

5. The method of claim 1 , wherein the second face resampling keyframe includes a second head transform matrix and second face landmark vertices, and further comprising determining a second interpolated face frame by calculating a respective displacement for each vertex in the second face landmark vertices.

6. The method of claim 1 , wherein the interpolating the first background and the second background is done with alpha blending.

7. The method of claim 1 , wherein the rendering includes at least one of feathering of edges of the interpolated first face and the interpolated second face or fading between a first interpolated face frame and a second interpolated face frame based on the interpolation amount.

8. The method of claim 1 , wherein:
the first face resampling keyframe includes a first head transform matrix and first face landmark vertices;
performing face tracking on the first video frame further includes determining first texture coordinates for the first face resampling keyframe and a timestamp; and
the first texture coordinates are applied to the first face landmark vertices.

9. The method of claim 1 , wherein the first face resampling keyframe is identified by performing red green blue (RGB) face tracking on the first video frame.

10. A non-transitory computer-readable medium with instructions stored thereon that, when executed by one or more computers, cause the one or more computers to perform operations, the operations comprising:
receiving a set of video frames that correspond to a video, the set of video frames including a first video frame and a second video frame that each include a face, wherein the second video frame is subsequent to the first video frame;
performing face tracking on the first video frame to identify a first face resampling keyframe;
performing face tracking on the second video frame to identify a second face resampling keyframe;
deriving an interpolation amount;
interpolating a first background of the first face resampling keyframe and a second background of the second face resampling keyframe based on the interpolation amount;
rendering an interpolated first face and an interpolated second face; and
displaying a final frame that is based on the interpolated first background, the interpolated second background, the interpolated first face, and the interpolated second face.

11. The computer-readable medium of claim 10 , wherein the operations further comprise:
blending the first background with the second background to obtain a blended background;
blending the interpolated first face with the interpolated second face to obtain a blended interpolated face; and
generating the final frame by placing a smooth face on top of the blended interpolated face and the blended background.

12. The computer-readable medium of claim 10 , wherein the first face resampling keyframe includes a first head transform matrix and first face landmark vertices and further comprising determining a first interpolated face frame by:
using the first head transform matrix to extract a translation vector, a rotation quaternion, and a scale vector;
linearly interpolating the translation vector;
using a linear interpolation to interpolate the rotation quaternion to generate an interpolated rotation quaternion;
linearly interpolating the scale vector to generate an interpolated scale vector;
composing an interpolated translation-rotation-scale matrix based on the interpolated translation vector, the interpolated rotation quaternion, and the interpolated scale vector; and
calculating an interpolated position for the interpolated first face using the interpolated translation-rotation-scale matrix.

13. The computer-readable medium of claim 10 , wherein the interpolation amount is derived from (a) a duration between the first face resampling keyframe and the second face resampling keyframe and (b) a current render time.

14. The computer-readable medium of claim 10 , wherein the second face resampling keyframe includes a second head transform matrix and second face landmark vertices, and the operations further include determining a second interpolated face frame by calculating a respective displacement for each vertex in the second face landmark vertices.

15. The computer-readable medium of claim 10 , wherein interpolating the first background and the second background is done with alpha blending.

16. A system comprising:
one or more processors; and
a memory that stores instructions that, when executed by the one or more processors cause the one or more processors to perform operations comprising:
receiving a set of video frames that correspond to a video, the set of video frames including a first video frame and a second video frame that each include a face, wherein the second video frame is subsequent to the first video frame;
performing face tracking on the first video frame to identify a first face resampling keyframe;
performing face tracking on the second video frame to identify a second face resampling keyframe;
deriving an interpolation amount;
interpolating a first background of the first face resampling keyframe and a second background of the second face resampling keyfr

Assignees

• Google Llc

Inventors

• Cower Dillon

Classifications

• G06V40/172 Primary

  Classification, e.g. identification · CPC title

• G06T3/4007

  based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403) · CPC title

• G06T5/002

  Physics · mapped topic

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T7/248

  involving reference images or patches · CPC title