Super-resolution processing method for moving image and image processing apparatus therefor

Patent metadata

Field	Value
Publication number	US-2018338159-A1
Application number	US-201815982382-A
Country	US
Kind code	A1
Filing date	May 17, 2018
Priority date	May 17, 2017
Publication date	Nov 22, 2018
Grant date	—

Abstract

A super-resolution processing method of a moving image is provided. The super-resolution processing method of a moving image includes sequentially inputting a plurality of input frames included in the video to any one of a recurrent neural network (RNN) for super-resolution processing and a convolutional neural network (CNN) for super-resolution processing, sequentially inputting a frame sequentially output from the any one of the RNN and the CNN to an additional one of the RNN and the CNN, and upscaling a resolution of the output frame by carrying out deconvolution with respect to a frame sequentially output from the additional one of the RNN and the CNN.

First claim

What is claimed is:

1 . A super-resolution processing method of a video, the method comprising:
sequentially inputting a plurality of frames included in the video to a recurrent neural network (RNN) for super-resolution processing or a convolutional neural network (CNN) for super-resolution processing; in response to a plurality of frames being sequentially output from the RNN or the CNN, sequentially inputting the plurality of frames to the CNN or the RNN, respectively; and upscaling a resolution of a plurality of frames output from the CNN or the RNN by performing deconvolution with respect to the plurality of frames output from the CNN or the RNN.

2 . The super-resolution processing method as claimed in claim 1 , wherein sequentially inputting the plurality of frames included in the video to the RNN or the CNN further comprises sequentially inputting the plurality of frames to the RNN, and
wherein sequentially inputting the plurality of frames sequentially output from the RNN or the CNN to the CNN or the RNN further comprises sequentially inputting a plurality of frames output from the RNN to the CNN.

3 . The super-resolution processing method as claimed in claim 2 , further comprising:
recurring a plurality of frames sequentially output from the CNN to the RNN, wherein a plurality of frames output from the CNN include information relating to a frame of higher resolution than a corresponding frame input to the RNN.

4 . The super-resolution processing method as claimed in claim 2 , further comprising:
sequentially inputting a plurality of frames sequentially output from the CNN to an additional RNN; and recurring information relating to a hidden status of the additional RNN to the RNN, wherein the upscaling comprises performing deconvolution for a plurality of frames sequentially output from the additional RNN.

5 . The super-resolution processing method as claimed in claim 2 , wherein the CNN generates a feature map by filtering out a plurality of frames sequentially output from the RNN, performing batch normalization with respect to the feature map, and applying an activation function to the normalized feature map.

6 . The super-resolution processing method as claimed in claim 1 , further comprising:
predicting a scene change of the video by using information on a hidden status of a previous frame recurrent from the RNN; based on the scene change being predicted, changing information on the hidden status of the previous frame to a zero value; and updating information on a hidden status of a current frame based on the information on the hidden state of the previous frame changed to the zero value.

7 . The super-resolution processing method as claimed in claim 6 , wherein predicting the scene change further comprises calculating an error rate by using the information on a hidden status of a current frame predicted based on the information on the hidden status of the previous frame, and
predicting a scene change of the video according to whether the calculated error rate exceeds a predetermined threshold.

8 . The super-resolution processing method as claimed in claim 1 , wherein sequentially inputting the plurality of frames included in the video to the RNN or the CNN further comprises, based on an output format of the video being a YCbCr channel, sequentially inputting only frames corresponding to a Y channel from among the plurality of frames to the RNN or the CNN.

9 . The super-resolution processing method as claimed in claim 1 , wherein the RNN includes a long short-term memory (LSTM) network.

10 . An image processing apparatus which carries out super-resolution processing of a video, comprising:
an inputter configured to receive input of the video; and a processor configured to: sequentially input a plurality of frames included in the video to a recurrent neural network (RNN) for super-resolution processing and a convolutional neural network (CNN) for super-resolution processing, in response to a plurality of frames being sequentially output from the RNN or the CNN, sequentially input the plurality of frames to the CNN or the RNN, respectively, and upscale a resolution of a plurality of frames output from the CNN or the RNN by performing deconvolution for the plurality of frames sequentially output from the CNN or the RNN.

11 . The image processing apparatus as claimed in claim 10 , wherein the processor is further configured to:
sequentially input the plurality of frames to the RNN, and sequentially input the plurality of frames output from the RNN to the CNN.

12 . The image processing apparatus as claimed in claim 11 , wherein the processor is further configured to recur a plurality of frames sequentially output from the CNN to the RNN, and
wherein a plurality of frames output from the CNN include information on a frame of higher resolution than a corresponding frame input to the RNN.

13 . The image processing apparatus as claimed in claim 11 , wherein the processor is further configured to:
sequentially input a plurality of frames sequentially output from the CNN to an additional RNN, recur information on a hidden status of the additional RNN to the RNN, and carry out deconvolution for a plurality of frames sequentially output from the additional RNN.

14 . The image processing apparatus as claimed in claim 11 , wherein the CNN generates a feature map by filtering out a frame sequentially input from the RNN, carries out batch normalization for the feature map, and applies an activation function to the normalized feature map.

15 . The image processing apparatus as claimed in claim 11 , wherein the processor is further configured to:
predict a scene change of the video by using information on a hidden status of a previous frame recurrent from the RNN, based on the scene change being predicted, change the information on the hidden status of the previous frame to a zero value, and update information on a hidden status of a current frame based on the information on the hidden status of the previous frame changed to a zero value.

16 . The image processing apparatus as claimed in claim 15 , wherein the processor is further configured to:
calculate an error rate by using the information on a hidden state of a current frame predicted based on the information on the hidden status of the previous frame, and predict a scene change of the video according to whether the calculated error rate exceeds a predetermined threshold.

17 . The image processing apparatus as claimed in claim 10 , wherein the processor is further configured to, based on an output format of the video being a YCbCr channel, sequentially input only frames corresponding to a Y channel from among the plurality of frames to the RNN or the CNN.

18 . The image processing apparatus as claimed in claim 10 , wherein the RNN includes a long short-term memory (LSTM) network.

19 . A recording medium that stores a program for executing a super-resolution processing method of a video, wherein the super-resolution processing method comprises:
sequentially inputting a plurality of frames included in the video to a recurrent neural network (RNN) for super-resolution processing or a convolutional neural network (CNN) for super-resolution processing; in response to a plurality of frames being sequentially output from the RNN or the CNN, sequentially inputting the plurality of frames to the CNN or the RNN, respectively; and upscaling a resolution of a plurality of frames output from the CNN or the RNN by performin

Assignees

• Samsung Electronics Co Ltd

Inventors

• Kapoor Parichay
• Oh Ji-Hun
• Son Kwang-Hoon
• Cho Young-Chul
• Park Yong-Sup
• Kim Hyun-Jung

Classifications

• H04N23/10

  for generating image signals from different wavelengths · CPC title

• G06N3/045

  Combinations of networks · CPC title

• H04N23/6811

  based on the image signal · CPC title

• G06N3/048

  Activation functions · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title