Video encoding by providing geometric proxies

Patent metadata

Field	Value
Publication number	US-12192518-B2
Application number	US-202117443913-A
Country	US
Kind code	B2
Filing date	Jul 28, 2021
Priority date	Sep 26, 2018
Publication date	Jan 7, 2025
Grant date	Jan 7, 2025

Abstract

Compressing a frame of video includes receiving a frame of a video, identifying a three dimensional (3D) object in the frame, matching the 3D object to a stored 3D object, compressing the frame of the video using a color prediction scheme based on the 3D object and the stored 3D object, and storing the compressed frame with metadata, the metadata identifying the 3D object, indicating a position of the 3D object in the frame of the video and indicating an orientation of the 3D object in the frame of the video.

First claim

What is claimed is:

1. A method comprising:
receiving a frame of a video;
identifying a three-dimensional (3D) object in the frame;
matching the 3D object to a stored 3D object;
generating a first 3D object proxy based on the stored 3D object;
transforming the first 3D object proxy based on the 3D object identified in the frame;
compressing the frame of the video using a color prediction scheme based on the transformed first 3D object proxy and a transformed second 3D object proxy based on the stored 3D object; and
storing the compressed frame with metadata, the metadata identifying the 3D object, indicating a position of the 3D object in the frame of the video and indicating an orientation of the 3D object in the frame of the video.

2. The method of claim 1 , wherein the compressing of the frame of the video using the color prediction scheme based on the 3D object and the stored 3D object includes:
generating the second 3D object proxy based on the stored 3D object;
identifying the 3D object in a key frame of the video;
transforming the second 3D object proxy based on the 3D object identified in the key frame;
mapping color attributes from the 3D object to the transformed first 3D object proxy;
mapping color attributes from the 3D object identified in the key frame to the transformed second 3D object proxy; and
generating residuals for the 3D object based on the color attributes for the transformed first 3D object proxy and the color attributes for the transformed second 3D object proxy.

3. The method of claim 1 , wherein the compressing of the frame of the video using the color prediction scheme based on the 3D object and the stored 3D object includes:
generating the second 3D object proxy based on the stored 3D object;
identifying the 3D object in a key frame of the video;
transforming the second 3D object proxy based on the 3D object identified in the key frame;
mapping color attributes from the 3D object to the transformed first 3D object proxy; and
generating residuals for the 3D object based on the color attributes for the transformed first 3D object proxy and default color attributes for the transformed second 3D object proxy.

4. The method of claim 1 , wherein the compressing of the frame of the video using the color prediction scheme based on the 3D object and the stored 3D object includes:
encoding the first 3D object proxy using an auto encoder, wherein the transforming the first 3D object proxy includes transforming the encoded first 3D object proxy based on the 3D object identified in the frame;
generating the second 3D object proxy based on the stored 3D object;
encoding the second 3D object proxy using an autoencoder;
identifying the 3D object in a key frame of the video;
transforming the encoded second 3D object proxy based on the 3D object identified in the key frame;
mapping color attributes from the 3D object to the transformed first 3D object proxy;
mapping color attributes from the 3D object identified in the key frame to the transformed second 3D object proxy; and
generating residuals for the 3D object based on the color attributes for the transformed first 3D object proxy and the color attributes for the transformed second 3D object proxy.

5. The method of claim 1 , wherein the compressing of the frame of the video using the color prediction scheme based on the 3D object and the stored 3D object includes:
encoding the first 3D object proxy using an auto encoder, wherein the transforming the first 3D object proxy includes transforming the encoded first 3D object proxy based on the 3D object identified in the frame;
generating the second 3D object proxy based on the stored 3D object;
encoding the second 3D object proxy using an autoencoder;
identifying the 3D object in a key frame of the video;
transforming the encoded second 3D object proxy based on the 3D object identified in the key frame;
mapping color attributes from the 3D object to the transformed first 3D object proxy; and
generating residuals for the 3D object based on the color attributes for the transformed first 3D object proxy and default color attributes for the transformed second 3D object proxy.

6. The method of claim 1 , further comprising:
before storing the 3D object:
identifying at least one 3D object of interest associated with the video;
determining a plurality of mesh attributes associated with the 3D object of interest;
determining a position associated with the 3D object of interest;
determining an orientation associated with the 3D object of interest;
determining a plurality of color attributes associated with the 3D object of interest; and
reducing a number of variables associated with the mesh attributes for the 3D object of interest using an autoencoder.

7. The method of claim 1 , wherein compressing the frame of the video includes determining position coordinates of the 3D object relative to an origin coordinate of a background 3D object in a key frame.

8. The method of claim 1 , wherein
the stored 3D object includes default color attributes, and
the color prediction scheme uses the default color attributes.

9. The method of claim 1 , further comprising:
identifying at least one 3D object of interest associated with the video;
generating at least one stored 3D object based on the at least one 3D object of interest, each of the at least one stored 3D object being defined by a mesh including a collection of points connected by faces, each point storing at least one attribute, the at least one attribute including a position coordinate for the respective point; and
storing the at least one stored 3D object in association with the video.

10. A method comprising:
receiving a frame of a video;
identifying a three-dimensional (3D) object in the frame;
matching the 3D object to a stored 3D object;
generating a 3D object proxy based on the stored 3D object;
transforming the 3D object proxy based on the 3D object identified in the frame;
mapping color attributes from the 3D object to the transformed 3D object proxy;
compressing the frame of the video using a color prediction scheme based on the transformed 3D object proxy, a second transformed 3D object proxy and the stored 3D object; and
storing the frame of the video.

11. The method of claim 10 , wherein the 3D object proxy is a first 3D object proxy and the compressing of the frame of the video using the color prediction scheme based on the 3D object and the stored 3D object includes:
generating the second 3D object proxy based on the stored 3D object;
identifying the 3D object in a key frame of the video;
transforming the second 3D object proxy as the second transformed 3D object proxy based on the 3D object identified in the key frame;
mapping color attributes from the 3D object identified in the key frame to the transformed second 3D object proxy; and
generating color attributes for the 3D object based on the color attributes for the transformed first 3D object proxy and the color attributes for the transformed second 3D object proxy.

12. The method of claim 10 , wherein the 3D object proxy is a first 3D object proxy and the compressing of the frame of the video using the color prediction scheme based on the 3D object and the stored 3D object includes:
encoding the first 3D object proxy using an autoencoder;
transforming the encoded first 3D object proxy based on metadata associated with the 3D object;
generating the second 3D object proxy based on the stored 3D object;
encoding the second 3D object proxy using an autoencoder;
identifying the 3D object in a key frame of the video;
transforming the encoded second 3D object proxy

Assignees

• Google Llc

Inventors

• Hemmer Michael
• Makadia Ameesh

Classifications

• H04N19/42

  characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation (H04N19/635 takes precedence) · CPC title

• H04N19/186

  the unit being a colour or a chrominance component · CPC title

• H04N19/593

  involving spatial prediction techniques · CPC title

• H04N19/20

  using video object coding · CPC title

• G06T9/004

  Predictors, e.g. intraframe, interframe coding · CPC title