Apparatus and method for performing scalable video decoding

Patent metadata

Field	Value
Publication number	US-11610283-B2
Application number	US-202016831805-A
Country	US
Kind code	B2
Filing date	Mar 27, 2020
Priority date	Mar 28, 2019
Publication date	Mar 21, 2023
Grant date	Mar 21, 2023

Abstract

Provided are a method and an apparatus for performing scalable video decoding, wherein the method and the apparatus down-sample input video, determine the down-sampled input video as base layer video, generate prediction video for enhancement layer video by applying an up-scaling filter to the base layer video, and code the base layer video and the prediction video, wherein the up-scaling filter is a convolution filter of a deep neural network.

First claim

What is claimed is:

1. A method of performing scalable video decoding, the method comprising:
generating base layer video by down-sampling input video;
generating prediction video for enhancement layer video by selectively applying a fixed up-scaling filter, which has fixed filter coefficient values, and a convolution filter of a deep neural network to the base layer video; and
coding the base layer video and the prediction video,
wherein the generating of the prediction video for the enhancement layer video comprises generating the prediction video for the enhancement layer video by applying one convolution filter corresponding to a compression and distortion degree of the generated base layer video to the base layer video among a plurality of convolution filters of a plurality of deep neural networks which are pre-trained according to a plurality of compression and distortion degrees.

2. The method of claim 1 , wherein the generating of the prediction video comprises generating the prediction video for the enhancement layer video by applying a bi-cubic interpolation to a chrominance component of the base layer video and applying the convolution filter of the deep neural network to a luminance component of the base layer video.

3. The method of claim 1 , wherein the deep neural network is a deep neural network trained on the basis of a difference between video scaled up from low-resolution luminance input video and high-resolution original video.

4. The method of claim 1 , wherein the deep neural network comprises a plurality of residual blocks in which two convolution layers and two activation functions are alternately connected.

5. The method of claim 4 , wherein the activation functions comprise leaky rectified linear units (LReLUs).

6. The method of claim 1 , wherein the deep neural network comprises a pixel shuffle layer.

7. A non-transitory computer-readable recording medium recording thereon a program for executing the method of claim 1 in a computer.

8. An apparatus for performing scalable video decoding, the apparatus comprising:
a controller configured to generate base layer video by down-sampling input video, generate prediction video for enhancement layer video by selectively applying a fixed up-scaling filter, which has fixed filter coefficient values, and a convolution filter of a deep neural network to the base layer video, and code the base layer video and the prediction video,
wherein the controller is configured to generate the prediction video for the enhancement layer video by applying one convolution filter corresponding to a compression and distortion degree of the generated base layer video to the base layer video among a plurality of convolution filters of a plurality of deep neural networks which are pre-trained according to a plurality of compression and distortion degrees.

9. The apparatus of claim 8 , wherein the controller generates the prediction video for the enhancement layer video by applying a bi-cubic interpolation to a chrominance component of the base layer video and applying the convolution filter of the deep neural network to a luminance component of the base layer video.

10. The apparatus of claim 8 , wherein the deep neural network is a deep neural network trained on the basis of a difference between video scaled up from low-resolution luminance input video and high-resolution original video.

11. The apparatus of claim 8 , wherein the deep neural network comprises a plurality of residual blocks in which two convolution layers and two activation functions are alternately connected.

12. The apparatus of claim 11 , wherein the activation functions comprise leaky rectified linear units (LReLUs).

13. The apparatus of claim 8 , wherein the deep neural network comprises a pixel shuffle layer.

Assignees

• Agency Defense Dev

Inventors

• Jung Ju Hyun
• Kim Dong Hyun
• Park No Hyeok
• Choi Jeung Won
• Kim Dae Eun
• Ki Se Hwan
• Kim Mun Churl
• Jun Ki Nam
• Baek Seung Ho
• Ko Jong Hwan

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• H04N19/59

  involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution · CPC title

• H04N19/33 Primary

  in the spatial domain · CPC title

• G06N3/08

  Learning methods · CPC title

• H04N19/186

  the unit being a colour or a chrominance component · CPC title