Using residual video data resulting from a compression of original video data to improve a decompression of the original video data

Patent metadata

Field	Value
Publication number	US-11496773-B2
Application number	US-202117352064-A
Country	US
Kind code	B2
Filing date	Jun 18, 2021
Priority date	Nov 21, 2017
Publication date	Nov 8, 2022
Grant date	Nov 8, 2022

Abstract

A method, computer readable medium, and system are disclosed for identifying residual video data. This data describes data that is lost during a compression of original video data. For example, the original video data may be compressed and then decompressed, and this result may be compared to the original video data to determine the residual video data. This residual video data is transformed into a smaller format by means of encoding, binarizing, and compressing, and is sent to a destination. At the destination, the residual video data is transformed back into its original format and is used during the decompression of the compressed original video data to improve a quality of the decompressed original video data.

First claim

What is claimed is:

1. A method comprising:
at a device:
performing a video compression on an original frame of original video data to generate a compressed frame;
determining residual video data for the compressed frame by calculating a difference between the original frame and the compressed frame;
encoding the residual video data, by downsizing a feature map of the residual video data, to create encoded residual video data;
binarizing the encoded residual video data, by converting the downsized feature map into a binary feature map, to create binarized residual video data;
compressing the binarized residual video data to create compressed residual video data; and
streaming the compressed residual video data with the compressed frame in a streaming packet, wherein the compressed residual video data is included in a meta field of the streaming packet.

2. The method of claim 1 , wherein the encoding and the binarizing are performed by a domain-specific autoencoder.

3. The method of claim 2 , wherein the domain-specific autoencoder is a domain-specific binary residual autoencoder.

4. The method of claim 3 , wherein the domain-specific binary residual autoencoder includes a binarizer that transforms frame data of the encoded residual video data into a compressible binary bitstream.

5. The method of claim 4 , wherein the binarizer is trained and implemented utilizing a tanh activation.

6. The method of claim 4 , wherein the binarizer is trained and implemented utilizing a hard tanh activation.

7. The method of claim 4 , wherein the binarized residual video data is compressed utilizing the domain-specific binary residual autoencoder.

8. The method of claim 7 , wherein the domain-specific binary residual autoencoder includes a Huffman encoder that compresses the binary bitstream produced by the binarizer in order to reduce an amount of data to be transmitted.

9. The method of claim 2 , domain-specific autoencoder is trained using training data that is constrained to a specific domain.

10. The method of claim 9 , wherein the specific domain is a particular video game.

11. The method of claim 1 , wherein the streaming packet is streamed to a receiver.

12. The method of claim 1 , further comprising:
decompressing the compressed frame at a client device to create a decompressed frame;
reconstructing the compressed residual video data at the client device to create reconstructed residual video data; and
adding the reconstructed residual video data back to the decompressed frame to create output video data.

13. The method of claim 1 , wherein the video compression is a domain-specific video compression.

14. A system comprising:
a non-transitory memory storing instructions; and
one or more processors in communication with the non-transitory memory that execute the instructions to:
perform a video compression on an original frame of original video data to generate a compressed frame;
determine residual video data for the compressed frame by calculating a difference between the original frame and the compressed frame;
encode the residual video data, by downsizing a feature map of the residual video data, to create encoded residual video data;
binarize the encoded residual video data, by converting the downsized feature map into a binary feature map, to create binarized residual video data;
compress the binarized residual video data to create compressed residual video data; and
stream the compressed residual video data with the compressed frame in a streaming packet, wherein the compressed residual video data is included in a meta field of the streaming packet.

15. The system of claim 14 , wherein the encoding and the binarizing are performed by a domain-specific autoencoder.

16. The system of claim 15 , wherein the domain-specific autoencoder is a domain-specific binary residual autoencoder.

17. The system of claim 16 , wherein the domain-specific binary residual autoencoder includes a binarizer that transforms frame data of the encoded residual video data into a compressible binary bitstream.

18. The system of claim 17 , wherein the binarizer is trained and implemented utilizing a tanh activation.

19. A non-transitory computer-readable media storing computer instructions which when executed by one or more processors of a device cause the device to:
perform a video compression on an original frame of original video data to generate a compressed frame;
determine residual video data for the compressed frame by calculating a difference between the original frame and the compressed frame;
encode the residual video data, by downsizing a feature map of the residual video data, to create encoded residual video data;
binarize the encoded residual video data, by converting the downsized feature map into a binary feature map, to create binarized residual video data;
compress the binarized residual video data to create compressed residual video data; and
stream the compressed residual video data with the compressed frame in a streaming packet, wherein the compressed residual video data is included in a meta field of the streaming packet.

20. The non-transitory computer-readable media of claim 19 , wherein the encoding and the binarizing are performed by a domain-specific autoencoder.

Assignees

• Nvidia Corp

Inventors

• Tsai Yi-Hsuan
• Liu Ming-Yu
• Sun Deqing
• Yang Ming-Hsuan
• Kautz Jan

Classifications

• H04N19/46

  Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence) · CPC title

• H04N19/85 Primary

  using pre-processing or post-processing specially adapted for video compression · CPC title

• H04N19/436

  using parallelised computational arrangements · CPC title

• H04N19/91

  Entropy coding, e.g. variable length coding [VLC] or arithmetic coding · CPC title