Frame-level super-resolution-based video coding

Patent metadata

Field	Value
Publication number	US-11330280-B2
Application number	US-201917044560-A
Country	US
Kind code	B2
Filing date	Apr 12, 2019
Priority date	Apr 13, 2018
Publication date	May 10, 2022
Grant date	May 10, 2022

Abstract

A method of encoding a video frame into a bitstream is described including: applying a downsampling scheme to an original video frame of a first resolution for determining a first low-resolution, LR, original frame and one or more second original LR frames, the first LR original frame and the one or more second LR original frames being of a second resolution that is lower than the first resolution; partitioning the first LR original frame into first original blocks and partitioning the one or more second LR frames into second original blocks; using a prediction method for determining first predicted blocks, the first predicted blocks defining predictions for the first original blocks of the first LR original frame; determining first residual blocks based on the first predicted blocks and the first original blocks and, subsequently, determining first reconstructed blocks based on the first residual blocks and the first predicted blocks; determining second predicted blocks based on the first reconstructed blocks, the second predicted blocks defining predictions for the second original blocks of the one or more second LR original frames; determining second residual blocks based on the second predicted blocks and the second original blocks; and, transforming the first residual blocks and second residual blocks into a bitstream, the bitstream including metadata for signalling a decoder apparatus that during the encoding a downsampling scheme was used for downsampling the video frame.

First claim

The invention claimed is:

1. A method of encoding a video frame into a bitstream, the method comprising:
a processor of an encoding apparatus making a frame-level decision in-loop to apply a frame-level super resolution prediction mode of encoding to an original video frame of a first resolution, wherein the decision is made at the frame-level of one or more video frames, and wherein the frame-level super resolution prediction mode is carried out in-loop and comprises:
based on the decision, the processor applying a downsampling scheme to the original video frame for determining a first low-resolution (LR) original frame and one or more second original LR frames, the first LR original frame and the one or more second LR original frames being of a second resolution that is lower than the first resolution;
based on the decision, the processor partitioning the first LR original frame into first original blocks and partitioning the one or more second LR frames into second original blocks;
based on the decision, the processor using a prediction method for determining first predicted blocks, the first predicted blocks defining predictions for the first original blocks of the first LR original frame;
based on the decision, the processor determining first residual blocks based on the first predicted blocks and the first original blocks and, subsequently, determining first reconstructed blocks based on the first residual blocks and the first predicted blocks;
based on the decision, the processor determining second predicted blocks based on the first reconstructed blocks, the second predicted blocks defining predictions for the second original blocks of the one or more second LR original frames;
based on the decision, the processor determining second residual blocks based on the second predicted blocks and the second original blocks; and
based on the decision, the processor transforming the first residual blocks and second residual blocks into the bitstream, the bitstream including metadata for signalling a decoder apparatus that during the encoding a downsampling scheme was used for downsampling the video frame.

2. The method according to claim 1 , wherein the video frame is an I frame, a P frame, or a B frame, and wherein determining the first predicted blocks comprises:
using an intra-prediction method for determining the first predicted blocks when the video frame is an I frame; and
using an inter-prediction method for determining the first predicted blocks when the video frame is P frame or a B frame.

3. The method according to claim 1 , wherein the first reconstructed blocks define a first LR reconstructed frame, the first LR reconstructed frame defining a reference frame for determining the second predicted blocks.

4. The method according to claim 3 wherein a block-matching prediction method is used for determining the second predicted blocks, the block-matching prediction method being based on an intra region-based template matching method or an intra copy block method.

5. The method according to claim 1 , wherein the first residual blocks associated with the first LR original frame and second residual blocks associated with the one or more second LR original frames define a data structure, the data structure being associated with metadata for a sub Group of Pictures (sub-GOP).

6. The method according to claim 1 , wherein the downsampling scheme is a polyphase downsampling scheme, the downsampling scheme including a plurality of spatial downsampling lattices.

7. The method according to claim 1 , wherein the first LR original frame and the one or more second LR original frames are spatially multiplexed into a multiplexed video frame of the first resolution.

8. The method according to claim 1 , wherein the metadata includes a type of metadata, the type of metadata being at least one of: (i) one or more first parameters for signalling the decoder apparatus the type of downsampling scheme that was used by the encoder apparatus; (ii) one or more second parameters for signalling the decoder apparatus the type of prediction that was used for predicting the first predicted blocks; or (iii) one or more third parameters for signalling the decoder apparatus the type of prediction that was used for predicting the second predicted blocks.

9. A method of decoding a bitstream into a video frame, the method comprising:
a processor of a decoding apparatus receiving the bitstream, the bitstream comprising first encoded residual blocks associated with a first low-resolution (LR) original frame and second encoded residual blocks associated with one or more second LR original frames;
the processor receiving metadata associated with the first and second encoded residual blocks, the metadata signalling the decoder apparatus that an encoder apparatus used a downsampling scheme of a frame-level super resolution prediction mode, carried out in-loop, for downsampling an original frame of a first resolution into the first LR original frame and the one or more second LR original frames, the first LR original frame and the one or more second LR original frames being of a second resolution that is lower than the first resolution;
based on the signalling, the processor applying a frame-level super resolution prediction mode of decoding to the bitstream, wherein the frame-level super resolution prediction mode is carried out in-loop and comprises:
based on the signalling, the processor using a prediction method for determining first predicted blocks, the first predicted blocks defining predictions for the first original blocks of the first LR original frame;
based on the signalling, the processor transforming the first encoded residual blocks into first residual blocks and determining first reconstructed blocks based on the first residual blocks and the first predicted blocks, the first reconstructed blocks defining a first LR reconstructed frame;
based on the signalling, the processor determining second predicted blocks based the first reconstructed blocks, the second predicted blocks defining predictions for the second original blocks of the one or more second LR original frames;
based on the signalling, the processor transforming the second encoded residual blocks into second residual blocks and determining second reconstructed blocks based on the second residual blocks and the second predicted blocks, the second reconstructed blocks defining one or more second LR reconstructed frames; and
based on the signalling, the processor determining a reconstructed original video frame based on the first LR reconstructed frame and the one or more second LR reconstructed frames.

10. The method according to claim 9 , wherein determining the reconstructed blocks comprises:
upsampling the first LR reconstructed frame and the one or more second LR reconstructed frames based on an upsampling scheme that is associated with the downsampling scheme; and
combining the upsampled first LR reconstructed frame and the upsampled second LR reconstructed frame into the reconstructed original video frame.

11. The method according to claim 9 , wherein a block-matching prediction method is used for determining the second predicted blocks, the block-matching prediction method being based on an intra region-based template matching method or a intra copy block method.

12. The method according to claim 1 , wherein the bitstream is at least one of a High Efficiency Video Coding (HVEC) based bitstream, an Alliance for Open Media Video (AVI) based bitstream, or a Versatile Video Coding (VVC) based bitstream.

13. An apparatus for encoding a video frame into a bitstream, the apparatus comprising:
a processor and m

Assignees

• Koninklijke Kpn Nv
• TNO

Inventors

• Da Silva Pratas Gabriel Alexandre
• Thomas Emmanuel

Classifications

• H04N19/46

  Embedding additional information in the video signal during the compression process (H04N19/517, H04N19/68, H04N19/70 take precedence) · CPC title

• H04N19/119

  Adaptive subdivision aspects, e.g. subdivision of a picture into rectangular or non-rectangular coding blocks · CPC title

• H04N19/176

  the region being a block, e.g. a macroblock · CPC title

• H04N19/159

  Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction · CPC title

• H04N19/147

  according to rate distortion criteria (rate-distortion as a criterion for motion estimation H04N19/567) · CPC title