Using residual video data resulting from a compression of original video data to improve a decompression of the original video data

Patent metadata

Field	Value
Publication number	US-11082720-B2
Application number	US-201816191174-A
Country	US
Kind code	B2
Filing date	Nov 14, 2018
Priority date	Nov 21, 2017
Publication date	Aug 3, 2021
Grant date	Aug 3, 2021

Abstract

A method, computer readable medium, and system are disclosed for identifying residual video data. This data describes data that is lost during a compression of original video data. For example, the original video data may be compressed and then decompressed, and this result may be compared to the original video data to determine the residual video data. This residual video data is transformed into a smaller format by means of encoding, binarizing, and compressing, and is sent to a destination. At the destination, the residual video data is transformed back into its original format and is used during the decompression of the compressed original video data to improve a quality of the decompressed original video data.

First claim

What is claimed is:

1. A method comprising:
at a device:
performing a domain-specific video compression on an original frame of original video data to generate a compressed frame;
determining residual video data for the compressed frame by calculating a difference between the original frame and the compressed frame;
using a domain-specific autoencoder:
encoding the residual video data, by downsizing a feature map of the residual video data, to create encoded residual video data, and
binarizing the encoded residual video data, by converting the downsized feature map into a binary feature map, to create binarized residual video data;

compressing the binarized residual video data to create compressed residual video data; and
streaming the compressed residual video data with the compressed frame in a streaming packet, wherein the compressed residual video data is included in a meta field of the streaming packet.

2. The method of claim 1 , wherein the domain-specific autoencoder is a domain-specific binary residual autoencoder.

3. The method of claim 2 , wherein the domain-specific binary residual autoencoder includes a binarizer that transforms frame data of the encoded residual video data into a compressible binary bitstream.

4. The method of claim 3 , wherein the binarizer is trained and implemented utilizing a tan h activation.

5. The method of claim 3 , wherein the binarizer is trained and implemented utilizing a hard tan h activation.

6. The method of claim 3 , wherein the binarized residual video data is compressed utilizing the domain-specific binary residual autoencoder.

7. The method of claim 6 , wherein the domain-specific binary residual autoencoder includes a Huffman encoder that compresses the binary bitstream produced by the binarizer in order to reduce an amount of data to be transmitted.

8. The method of claim 1 , wherein the streaming packet is streamed to a receiver.

9. The method of claim 1 , further comprising:
decompressing the compressed frame at a client device to create a decompressed frame;
reconstructing the compressed residual video data at the client device to create reconstructed residual video data; and
adding the reconstructed residual video data back to the decompressed frame to create output video data.

10. A system comprising:
a non-transitory memory storing instructions; and
one or more processors in communication with the non-transitory memory that execute the instructions to:
perform a domain-specific video compression on an original frame of original video data to generate a compressed frame;
determine residual video data for the compressed frame by calculating a difference between the original frame and the compressed frame;
use a domain-specific autoencoder to:
encode the residual video data, by downsizing a feature map of the residual video data, to create encoded residual video data, and
binarize the encoded residual video data, by converting the downsized feature map into a binary feature map, to create binarized residual video data;

compress the binarized residual video data to create compressed residual video data; and
stream the compressed residual video data with the compressed frame in a streaming packet, wherein the compressed residual video data is included in a meta field of the streaming packet.

11. The system of claim 10 , wherein the domain-specific autoencoder is a domain-specific binary residual autoencoder.

12. The system of claim 11 , wherein the domain-specific binary residual autoencoder includes a binarizer that transforms frame data of the encoded residual video data into a compressible binary bitstream.

13. The system of claim 12 , wherein the binarizer is trained and implemented utilizing a tan h activation.

14. A non-transitory computer-readable media storing computer instructions which when executed by one or more processors of a device cause the device to:
perform a domain-specific video compression on an original frame of original video data to generate a compressed frame;
determine residual video data for the compressed frame by calculating a difference between the original frame and the compressed frame;
use a domain-specific autoencoder to:
encode the residual video data, by downsizing a feature map of the residual video data, to create encoded residual video data, and
binarize the encoded residual video data, by converting the downsized feature map into a binary feature map, to create binarized residual video data;

compress the binarized residual video data to create compressed residual video data; and
stream the compressed residual video data with the compressed frame in a streaming packet, wherein the compressed residual video data is included in a meta field of the streaming packet.

15. The method of claim 1 , domain-specific autoencoder is trained using training data that is constrained to a specific domain.

16. The method of claim 15 , wherein the specific domain is a particular video game.

Assignees

• Nvidia Corp

Inventors

• Tsai Yi-Hsuan
• Liu Ming-Yu
• Sun Deqing
• Yang Ming-Hsuan
• Kautz Jan

Classifications

• H04N19/436

  using parallelised computational arrangements · CPC title

• H04N19/91

  Entropy coding, e.g. variable length coding [VLC] or arithmetic coding · CPC title

• H04N19/46

  Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence) · CPC title

• H04N19/85 Primary

  using pre-processing or post-processing specially adapted for video compression · CPC title