Feature-Domain Residual for Video Coding for Machines

Patent metadata

Field	Value
Publication number	US-2021314573-A1
Application number	US-202117218967-A
Country	US
Kind code	A1
Filing date	Mar 31, 2021
Priority date	Apr 7, 2020
Publication date	Oct 7, 2021
Grant date	—

Abstract

An apparatus includes at least one processor; and at least one non-transitory memory including computer program code; wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to: decode encoded data to generate decoded data, the encoded data having a bitrate lower than that of original data, and extract features from the decoded data; decode encoded residual features to generate decoded residual features; and generate enhanced decoded features as a result of combining the decoded residual features with the features extracted from the decoded data.

First claim

1 . An apparatus comprising:
at least one processor; and at least one non-transitory memory including computer program code; wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to: encode original data with a first codec to generate encoded data with a bitrate lower than that of the original data, and decoded data; encode the original data with at least one second learned codec to generate encoded residual features and decoded residual features; and generate enhanced decoded features as a result of combining the decoded residual features with features extracted from the decoded data generated with the first codec.

2 . The apparatus of claim 1 , wherein at least one machine processes or analyzes the decoded data using the enhanced decoded features.

3 . The apparatus of claim 1 , wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus at least to:
generate enhanced decoded video resulting from combining the decoded data with the enhanced decoded features; wherein at least one machine processes or analyzes the decoded data using the enhanced decoded video.

4 . The apparatus of claim 3 , wherein the enhanced decoded video is generated using a neural network.

5 . The apparatus of claim 1 , wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus at least to:
rather than generating the enhanced decoded features, generate enhanced decoded video resulting from combining the decoded data with the decoded residual features; wherein at least one machine processes or analyzes the decoded data using the enhanced decoded video.

6 . The apparatus of claim 5 , wherein the enhanced decoded video is generated using a neural network.

7 . The apparatus of claim 1 , wherein the residual features are encoded using at least one neural network, and the residual features are decoded using at least one neural network.

8 . The apparatus of claim 1 , wherein the features extracted from the decoded data generated with the first codec are extracted using a neural network.

9 . The apparatus of claim 1 , wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus at least to:
extract features from the original data; extract features from the decoded data; and generate the residual features, prior to being encoded, as a result of computing a difference between the features extracted from the decoded data and the features extracted from the original data.

10 .- 20 . (canceled)

21 . An apparatus comprising:
at least one processor; and at least one non-transitory memory including computer program code; wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to: decode encoded data to generate decoded data, the encoded data having a bitrate lower than that of original data, and extract features from the decoded data; decode encoded residual features to generate decoded residual features; and generate enhanced decoded features as a result of combining the decoded residual features with the features extracted from the decoded data.

22 . The apparatus of claim 21 , wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus at least to:
process or analyze the enhanced decoded features using at least one task neural network.

23 . The apparatus of claim 21 , wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus at least to:
generate enhanced decoded video as a result of combining the decoded data with the enhanced decoded features; wherein the combining of the decoded data with the enhanced decoded features to generate the enhanced decoded video is performed using a neural network; and process or analyze the enhanced decoded video using at least one task neural network.

24 . The apparatus of claim 21 , wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus at least to:
generate enhanced decoded video as a result of combining the decoded data with the decoded residual features; wherein the combining of the decoded data with the decoded residual features to generate the enhanced decoded video is performed using a neural network; and process or analyze the enhanced decoded video using at least one task neural network.

25 . The apparatus of claim 21 ,
wherein the features are extracted from the decoded data using a neural network; and the encoded residual features are decoded using a neural network.

26 . The apparatus of claim 21 , wherein the combining of the decoded residual features with the features extracted from the decoded data to generate the enhanced decoded features is a summation of the decoded residual features and the features extracted from the decoded data.

27 . The apparatus of claim 21 , wherein the encoded residual features are a difference between features extracted from the original data, and features extracted from preliminary decoded data or the features extracted from the decoded data.

28 . The apparatus of claim 21 , wherein the decoded residual features are decoded using entropy decoding and dequantization.

29 . The apparatus of claim 21 , wherein the decoded residual features are decoded using an image of a video decoder, the decoding of the residual features comprising converting decoded feature map images to the decoded residual features.

30 . The apparatus of claim 21 , wherein the original data is video data.

31 . A method comprising:
decoding encoded data to generate decoded data, the encoded data having a bitrate lower than that of original data, and extracting features from the decoded data; decoding encoded residual features to generate decoded residual features; and generating enhanced decoded features as a result of combining the decoded residual features with the features extracted from the decoded data.

32 . (canceled)

Assignees

• Nokia Technologies Oy

Inventors

• Zhang Honglei
• Rezazadegan Tavakoli Hamed
• Cricri Francesco
• Hannuksela Miska Matias
• Aksu Emre
• Le Nam

Classifications

• H04N19/30

  using hierarchical techniques, e.g. scalability (H04N19/63 takes precedence) · CPC title

• H04N19/50 Primary

  using predictive coding (H04N19/61 takes precedence) · CPC title

• G06T9/002

  using neural networks · CPC title

• H04N19/436

  using parallelised computational arrangements · CPC title

• H04N19/146 Primary

  Data rate or code amount at the encoder output · CPC title