Image processing method and processing device

Patent metadata

Field	Value
Publication number	US-10430683-B2
Application number	US-201815936755-A
Country	US
Kind code	B2
Filing date	Mar 27, 2018
Priority date	Nov 9, 2017
Publication date	Oct 1, 2019
Grant date	Oct 1, 2019

Abstract

There are provided an image processing method and an image processing device. The image processing method comprises: acquiring an input image; acquiring a first noise image and a second noise image; executing image conversion processing on the input image with the first noise image using a generative neural network, to acquire a first output image; and executing high resolution conversion processing on the first output image with the second noise image using a super-resolution neural network, to acquire a second output image, wherein the first noise image is different from the second noise image.

First claim

We claim:

1. An image processing method, comprising:
acquiring an input image;
acquiring a first noise image and a second noise image;
executing image conversion processing on the input image with the first noise image using a generative neural network, to output the input image converted as a first output image; and
executing high resolution conversion processing on the first output image with the second noise image using a super-resolution neural network, to output the first output image converted as a second output image,
wherein the first noise image is different from the second noise image.

2. The image processing method according to claim 1 , wherein
the input image comprises a first color component, a second color component and a third color component;
the first noise image comprises N components, where N is an integer greater than or equal to 1;
an input to the generative neural network comprises the N components of the first noise image and the first color component, the second color component and the third color component of the input image; and
an output from the generative neural network is the first output image which comprises a first color component, a second color component, and a third color component.

3. The image processing method according to claim 1 , wherein the generative neural network is configured to comprise one or more down-scale units, one or more residual units and one or more up-scale units, wherein
the down-scale units each comprise a convolutional layer, a down-scale layer and an instance normalization layer which are sequentially connected;
the residual units each comprise a convolutional layer and an instance normalization layer which are sequentially connected; and
the up-scale units each comprise an up-scale layer, an instance normalization layer and a convolutional layer which are sequentially connected, wherein a number of the up-scale units is equal to a number of the down-scale units.

4. The image processing method according to claim 1 , wherein the second noise image comprises M components, where M is an integer greater than or equal to 1, and an input to the super-resolution neural network comprises the M components of the second noise image and a first color component, a second color component and a third color component of the first output image; and
an output from the super-resolution neural network is the second output image which comprises the first color component, the second color component and the third color component.

5. The image processing method according to claim 1 , wherein the super-resolution neural network is configured to comprise an enhancement unit and a transform unit which are sequentially connected, and executing high resolution conversion processing using the super-resolution neural network comprises:
executing up-scale processing on the first output image with the second noise image using the enhancement unit and outputting a first intermediate image which comprises a luminance component, a first color difference component, and a second color difference component; and
transforming the first intermediate image output from the enhancement unit into the second output image which comprises a first color component, a second color component and a third color component using the transform unit.

6. The image processing method according to claim 5 , wherein the enhancement unit comprises a first sub-network, a second sub-network and a third sub-network, wherein
an input to each of the sub-networks is the first output image and the second noise image; and
all of the sub-networks have the same structure, and comprise the same number of convolutional layers and the same number of enhancement layers.

7. The image processing method according to claim 1 , wherein the input image is a first training image, the first noise image is a first training noise image, and the first output image is a first training output image, and the image processing method further comprises:
acquiring a second training noise image; and
generating a second training output image according to the first training image and the second training noise image using the generative neural network; and
training the generative neural network based on the first training image, the first training output image and the second training output image.

8. The image processing method according to claim 7 , wherein the training the generative neural network comprises:
inputting the first training output image to a discriminative neural network to acquire a discrimination label indicating whether the first training output image has converted features; and calculating a loss value of the generative neural network and optimizing parameters of the generative neural network according to the first training image, the first training output image, the second training output image and the discrimination label, wherein
the calculating a loss value of the generative neural network comprises:
acquiring content features of the first training image, the first training output image and the second training output image, and acquiring style features of the first training output image and the second training output image;
calculating, in accordance with a first loss function, the loss value of the generative neural network according to the acquired content features and style features and the discrimination label of the first training output image; and
optimizing the parameters of the generative neural network according to the loss value of the generative neural network.

9. The image processing method according to claim 8 , wherein
the first loss function comprises a style difference loss function, and the calculating the loss value of the generative neural network comprises: calculating, in accordance with the style difference loss function, a style loss value of the generative neural network according to the style features of the first training output image and the style features of the second training output image; and
the first loss function further comprises a content loss function, and the calculating the loss value of the generative neural network comprises: calculating, in accordance with the content loss function, a content loss value of the generative neural network according to the content features of the first training image, the first training output image and the second training output image.

10. The image processing method according to claim 1 , wherein the first output image is a first sample image, and the image processing method further comprises:
acquiring a super-resolution training noise image;
extracting a low-resolution image from the first sample image as a super-resolution training image, wherein a resolution of the super-resolution training image is lower than that of the first sample image;
acquiring a second sample image according to the super-resolution training image and the super-resolution training noise image using the super-resolution neural network, wherein a resolution of the second sample image is equal to that of the first sample image; and
optimizing parameters of the super-resolution neural network by reducing a cost function of the super-resolution neural network according to the first sample image and the second sample image.

11. An image processing device, comprising:
one or more processor; and
one or more memory,
wherein the memory has computer readable instructions stored thereon, which when being executed by the one or more processor, for controlling the processor to be configured to:
configure a generative neural network to execute image conversion processing on an input image with a first noise image, to output the in

Assignees

• Boe Technology Group Co Ltd

Inventors

• Liu Hanwen
• Navarrete Michelini Pablo

Classifications

• G06T11/00 Primary

  Two-dimensional [2D] image generation · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06N3/047

  Probabilistic or stochastic networks · CPC title

• G06N3/045

  Combinations of networks · CPC title