Video lighting using depth and virtual lights

Patent metadata

Field	Value
Publication number	US-11776201-B2
Application number	US-202217722480-A
Country	US
Kind code	B2
Filing date	Apr 18, 2022
Priority date	Jul 8, 2019
Publication date	Oct 3, 2023
Grant date	Oct 3, 2023

Abstract

Implementations described herein relate to methods, systems, and computer-readable media to relight a video. In some implementations, a computer-implemented method includes receiving a plurality of frames of a video. Each video frame includes depth data and color data for a plurality of pixels. The method further includes segmenting each frame based on the depth data to classify each pixel as a foreground pixel or a background pixel. The method further includes setting depth value of each background pixel to a fixed depth value and applying a Gaussian filter to smooth depth value for the plurality of pixels. The method further includes calculating surface normals based on the depth values of the plurality of pixels. The method further includes rendering a relighted frame by adding a virtual light based on the surface normals and the color data.

First claim

What is claimed is:

1. A non-transitory computer-readable medium with instructions stored thereon that, when executed by one or more hardware processors, cause the one or more hardware processors to perform operations comprising:
receiving, at a receiving device that is in a video call with a sender device, a frame of a video from the sender device, wherein the frame includes depth data and color data for a plurality of pixels;
segmenting the frame based on the depth data to classify each pixel as a foreground pixel or a background pixel;
setting the depth value of each background pixel to a fixed depth value;
smoothing depth values of the plurality of pixels classified as the foreground pixel;
calculating surface normals based on the smoothed depth values of the plurality of pixels;
creating a three-dimensional (3D) mesh based on the surface normals; and
rendering a relighted frame by adding a virtual light based on the 3D mesh and the color data.

2. The non-transitory computer-readable medium of claim 1 , wherein the operations further comprise displaying the relighted frame on a display screen of the receiving device.

3. The non-transitory computer-readable medium of claim 1 , wherein the surface normals define a reflection direction.

4. The non-transitory computer-readable medium of claim 1 , wherein the fixed depth value is selected as a value that is far away from the depth values of the foreground pixels.

5. The non-transitory computer-readable medium of claim 1 , wherein adding the virtual light includes computing adjustments to color values of the plurality of pixels based on a type of the virtual light or a position of a light source of the virtual light.

6. The non-transitory computer-readable medium of claim 1 , wherein the virtual light is a ring light, and wherein adding the virtual light comprises relighting the frame such that objects in the foreground are illuminated and objects in the background appear dark.

7. The non-transitory computer-readable medium of claim 1 , wherein segmenting the frame includes applying a temporal low pass filter.

8. A computing device comprising:
one or more hardware processors; and
a memory coupled to the one or more hardware processors, with instructions thereon, that when executed by the one or more hardware processors to perform operations comprising:
receiving a frame of a video from a sender device that is in a video call with the computing device, wherein the frame includes depth data and color data for a plurality of pixels;
segmenting the frame based on the depth data to classify each pixel as a foreground pixel or a background pixel;
setting the depth value of each background pixel to a fixed depth value;

smoothing depth values of the plurality of pixels classified as the foreground pixel;
calculating surface normals based on the smoothed depth values of the plurality of pixels;
creating a three-dimensional (3D) mesh based on the surface normals; and
rendering a relighted frame by adding a virtual light based on the 3D mesh and the color data.

9. The computing device of claim 8 , wherein the operations further comprise displaying the relighted frame on a display screen of the receiving device.

10. The computing device of claim 8 , wherein the surface normals define a reflection direction.

11. The computing device of claim 8 , wherein the fixed depth value is selected as a value that is far away from the depth values of the foreground pixels.

12. The computing device of claim 8 , wherein adding the virtual light includes computing adjustments to color values of the plurality of pixels based on a type of the virtual light or a position of a light source of the virtual light.

13. The computing device of claim 8 , wherein the virtual light is a ring light, and wherein adding the virtual light comprises relighting the frame such that objects in the foreground are illuminated and objects in the background appear dark.

14. The computing device of claim 8 , wherein segmenting the frame includes applying a temporal low pass filter.

Assignees

• Google Llc

Inventors

• Zhou Guangyu
• Cower Dillon

Classifications

• G06T15/506 Primary

  Illumination models · CPC title

• G06T5/002

  Physics · mapped topic

• G06T5/20

  using local operators · CPC title

• G06T7/194

  involving foreground-background segmentation · CPC title

• G06T7/50

  Depth or shape recovery · CPC title