Cascaded Prediction-Transform Approach for Mixed Machine-Human Targeted Video Coding

Patent metadata

Field	Value
Publication number	US-2021218997-A1
Application number	US-202017137609-A
Country	US
Kind code	A1
Filing date	Dec 30, 2020
Priority date	Jan 10, 2020
Publication date	Jul 15, 2021
Grant date	—

Abstract

Data may be encoded to minimize distortion after decoding, but the quality required for presentation of the decoded data to a machine and the quality required for presentation to a human may be different. To accommodate different quality requirements, video data may be encoded to produce a first set of encoded data and a second set of encoded data, where the first set may be decoded for use by one of a machine consumer or a human consumer, and a combination of the first set and the second set may be decoded for use by the other of a machine consumer or a human consumer. The first and second set may be produced with a neural encoder and a neural decoder, and/or may be produced with the use of prediction and transform neural network modules. A human-targeted structure and a machine-targeted structure may produce the sets of encoded data.

First claim

What is claimed is:

1 . A method comprising:
encoding data to produce a first set of encoded data; encoding the data to produce a second set of encoded data; and at least one of:
storing the first set of encoded data and the second set of encoded data with a non-transitory memory, wherein the non-transitory memory is accessible to a decoder; or
transmitting the first set of encoded data and the second set of encoded data to the decoder.

2 . The method of claim 1 , wherein the encoding of the data to produce the first set of encoded data comprises:
neural encoding the data; quantizing the neural encoded data; and lossless encoding the quantized neural encoded data.

3 . The method of claim 1 , wherein the encoding of the data to produce the first set of encoded data comprises:
computing a residual of a portion of the data, wherein the computing of the residual is based on a prediction based on a previously decoded portion of the data and a compensation; transforming the computed residual; quantizing the transformed residual; and lossless encoding the quantized transformed residual.

4 . The method of claim 1 , wherein the encoding of the data to produce the second set of encoded data comprises:
neural encoding the data, wherein neural encoding the data comprises combination of machine-targeted features extracted from the data with output of initial layers of a human-targeted neural network; quantizing the neural encoded data; and lossless encoding the quantized neural encoded data.

5 . The method of claim 1 , wherein the encoding of the data to produce the second set of encoded data comprises:
computing a residual of a portion of the data, wherein the computing of the residual is based on a prediction based on a previously decoded portion of the data, a compensation, and machine-targeted features extracted from the data which are converted with a conversion neural network; transforming the computed residual; quantizing the transformed residual; and lossless encoding the quantized transformed residual.

6 . An apparatus comprising:
at least one processor; and at least one non-transitory memory and computer program code; wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to perform:
encode data to produce a first set of encoded data;
encode the data to produce a second set of encoded data; and
at least one of:
store the first set of encoded data and the second set of encoded data; or
transmit the first set of encoded data and the second set of encoded data to a decoder.

7 . The apparatus of claim 6 , wherein encoding the data to produce the first set of encoded data comprises the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to:
neural encode the data; quantize the neural encoded data; and lossless encode the quantized neural encoded data.

8 . The apparatus of claim 6 , wherein encoding the data to produce the first set of encoded data comprises the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to:
compute a residual of a portion of the data, wherein computing the residual is based on a prediction based on a previously decoded portion of the data and a compensation; transform the computed residual; quantize the transformed residual; and lossless encoding the quantized transformed residual.

9 . The apparatus of claim 6 , wherein encoding the data to produce the second set of encoded data comprises the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to:
neural encode the data, wherein neural encoding the data comprises combination of machine-targeted features extracted from the data with output of initial layers of a human-targeted neural network; quantize the neural encoded data; and lossless encode the quantized neural encoded data.

10 . The apparatus of claim 6 , wherein encoding the data to produce the second set of encoded data comprises the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to:
compute a residual of a portion of the data, wherein computing the residual is based on a prediction based on a previously decoded portion of the data, a compensation, and machine-targeted features extracted from the data which are converted with a conversion neural network; transform the computed residual; quantize the transformed residual; and lossless encode the quantized transformed residual.

11 . A method comprising:
determining whether a human agent or a computer agent will use decoded data; based on a determination that the computer agent will use decoded data, decoding a first set of encoded data to produce first data and providing the first data for the computer agent; and based on a determination that the human agent will use decoded data or a determination that the computer agent and the human agent will use decoded data, decoding a combination of the first set of encoded data and a second set of encoded data to produce second data and providing the second data for at least one of the human agent or the computer agent.

12 . The method of claim 11 , wherein the decoding of the first set of encoded data to produce the first data comprises:
lossless decoding the first set of encoded data; and inverse quantizing the lossless decoded first set of encoded data.

13 . The method of claim 11 , wherein the decoding of the combination of the first set of encoded data and the second set of encoded data to produce the second data comprises:
lossless decoding the second set of encoded data; inverse quantizing the lossless decoded second set of encoded data; inverse transforming the inverse quantized lossless decoded second set of encoded data; and compensating a combination of the inverse transformed inverse quantized lossless decoded second set of encoded data and machine-targeted features which are converted with a conversion neural network.

14 . The method of claim 11 , further comprising at least one of:
determining a first rate loss based, at least partially, on the first set of encoded data; transmitting the first data to one or more task neural networks and determining a respective task loss for the one or more task neural networks; determining a consumption loss based, at least partially, on the second video data; or determining a second rate loss based, at least partially, on the second set of encoded data.

15 . The method of claim 14 , further comprising at least one of:
causing training of at least one neural network used to encode the first set of encoded data based, at least partially, on the first rate loss; causing training of at least one neural network used to decode the first set of encoded data based, at least partially, on the first rate loss; causing training of the one or more task neural networks based, at least partially, on the first rate loss; causing training of the at least one neural network used to encode the first set of encoded data based, at least partially, on the one or more task losses; causing training of the at least one neural network used to decode the first set of encoded data based, at least partially, on the one or more task losses; causing training of the one or more task neural networks based, at least partially, one the one or more task losses; causing training of at leas

Assignees

• Nokia Technologies Oy

Inventors

• Rezazadegan Tavakoli Hamed
• Cricri Francesco
• Hannuksela Miska Matias
• Aksu Emre Baris
• Zhang Honglei
• Le Nam

Classifications

• H04N19/30 Primary

  using hierarchical techniques, e.g. scalability (H04N19/63 takes precedence) · CPC title

• H04N19/619 Primary

  the transform being operated outside the prediction loop · CPC title

• H04N19/192

  the adaptation method, adaptation tool or adaptation type being iterative or recursive · CPC title

• H04N19/176

  the region being a block, e.g. a macroblock · CPC title

• H04N19/134

  characterised by the element, parameter or criterion affecting or controlling the adaptive coding · CPC title