Apparatus for upscaling an image, method for training the same, and method for upscaling an image

Patent metadata

Field	Value
Publication number	US-10515435-B2
Application number	US-201715821095-A
Country	US
Kind code	B2
Filing date	Nov 22, 2017
Priority date	Apr 27, 2017
Publication date	Dec 24, 2019
Grant date	Dec 24, 2019

Abstract

The disclosure discloses an apparatus for upscaling an image, a method for training the same, and a method for upscaling an image, where a convolutional neural network circuit obtains feature images of the image, a multiplexer upscales the image by integrating every n*n feature images of an input signal into a feature image with a resolution which is n times the resolution of a feature image of the image, where n is an integer greater than 1.

First claim

The invention claimed is:

1. An apparatus for upscaling an image, comprising: at least one first convolutional neural network circuit and at least one multiplexer, which are connected in cascade, wherein:
one of the at least one first convolutional neural network circuit is located at a first level of the apparatus for upscaling an image, and a signal input terminal of the first convolutional neural network circuit located at the first level of the apparatus for upscaling an image is served as a signal input terminal of the apparatus for upscaling an image; one of the at least one multiplexer is located at a last level of the apparatus for upscaling an image, and a signal output terminal of the multiplexer located at the last level of the apparatus for upscaling an image is served as a signal output terminal of the apparatus for upscaling an image; and each multiplexer has a signal input terminal connected with a signal output terminal of one of the at least one first convolutional neural network circuit, or with a signal output terminal of another multiplexer;
each first convolutional neural network circuit is configured to convert an image of an input signal input to the first convolutional neural network circuit into a plurality of feature images, and to output the feature images; and
each multiplexer is configured to integrate every n*n feature images among feature images of an input signal input to the multiplexer into a feature image with a resolution which is n times a resolution of a feature image of the input signal input to the multiplexer, and to output the feature image, wherein a number of feature images of the input signal input to the multiplexer is a multiple of n*n, and n is an integer greater than 1;
wherein the apparatus for upscaling an image further comprises a second convolutional neural network circuit, wherein a signal input terminal of the second convolutional neural network circuit is connected with the signal output terminal of the multiplexer located at the last level of the apparatus for upscaling an image, and a signal output terminal of the second convolutional neural network circuit is served as the signal output terminal of the apparatus for upscaling an image; and
the second convolutional neural network circuit is configured to optimize a feature image of an output signal of the multiplexer located at the last level of the apparatus for upscaling an image.

2. The apparatus for upscaling an image according to claim 1 , wherein there are two or three multiplexers in the apparatus for upscaling an image.

3. The apparatus for upscaling an image according to claim 2 , wherein a signal input terminal of each multiplexer is connected with a signal output terminal of one of the at least one first convolutional neural network circuit.

4. The apparatus for upscaling an image according to claim 1 , wherein when there are a plurality of multiplexers, upscaling factors of respective multiplexers are same.

5. The apparatus for upscaling an image according to claim 1 , wherein each multiplexer is a multiplexer with an upscaling factor of a prime number.

6. The apparatus for upscaling an image according to claim 5 , wherein each multiplexer is a multiplexer with an upscaling factor of 2.

7. The apparatus for upscaling an image according to claim 1 , wherein each multiplexer is an adaptive interpolation filter.

8. The apparatus for upscaling an image according to claim 1 , wherein each convolutional neural network circuit of the at least one first convolutional neural network circuit and the second convolutional neural network circuit comprises at least one convolution layer comprising a plurality of filter circuits.

9. A method for training the apparatus for upscaling an image according to claim 1 , the method comprising:
initializing respective parameters of the apparatus for upscaling an image; and
taking an original image signal as an output signal of the apparatus for upscaling an image, taking an image signal which is obtained by downscaling the original image signal, as an input signal of the apparatus for upscaling an image, and adjusting the respective parameters of the apparatus for upscaling an image so that an image signal which is obtained by upscaling the downscaled image signal via respective adjusted parameters is same as the original image signal.

10. The training method according to claim 9 , wherein initializing the respective parameters of the apparatus for upscaling an image comprises:
initializing a weight W ij of each filter circuit at each convolution layer of each first and second convolutional neural network circuit in the apparatus for upscaling an image in an equation of:

W
ij

=

{

1
/

(
m
)

(

i
,
j

)

⁢

⁢
represents
⁢

⁢
a
⁢

⁢
set
⁢

⁢
anchor
⁢

⁢
pixel

0

other
⁢

⁢
pixels

}

;

wherein m represents a number of feature images input to a corresponding filter circuit; and
initializing a bias of each filter circuit to 0.

11. The training method according to claim 9 , wherein initializing the respective parameters of the apparatus for upscaling an image comprises:
initializing a weight W ij of each filter circuit at each convolution layer of each first and second convolutional neural network circuit in the apparatus for upscaling an image in an equation of:

W

Assignees

• Boe Technology Group Co Ltd

Inventors

• Navarrete Michelini Pablo
• Liu Hanwen
• Li Xiaoyu

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06T1/20

  Processor architectures; Processor configuration, e.g. pipelining · CPC title

• H04N19/132 Primary

  Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking · CPC title

• G06T3/4046 Primary

  using neural networks · CPC title

• G06T3/4007

  based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403) · CPC title