Image processing method and device, training method of neural network, image processing method based on combined neural network model, constructing method of combined neural network model, neural network processor, and storage medium

Patent metadata

Field	Value
Publication number	US-11954822-B2
Application number	US-202017419350-A
Country	US
Kind code	B2
Filing date	Oct 13, 2020
Priority date	Oct 18, 2019
Publication date	Apr 9, 2024
Grant date	Apr 9, 2024

Abstract

An image processing method, an image processing device, a training method of a neural network, an image processing method based on a combined neural network model, a constructing method of a combined neural network model, a neural network processor, and a storage medium are provided. The image processing method includes: obtaining, based on an input image, initial feature images of N stages with resolutions from high to low, where N is a positive integer and N>2, performing, based on initial feature images of second to N-th stages, cyclic scaling processing on an initial feature image of a first stage, to obtain an intermediate feature image; and preforming merging processing on the intermediate feature image to obtain an output image. The cyclic scaling processing includes hierarchically-nested scaling processing of N−1 stages, and scaling processing of each stage includes down-sampling processing, concatenating processing, up-sampling processing, and residual link addition processing.

First claim

What is claimed is:

1. An image processing method, comprising:
obtaining an input image;
obtaining, based on the input image, initial feature images of N stages with resolutions from high to low, wherein N is a positive integer and N>2;
performing, based on initial feature images of second to N-th stages, cyclic scaling processing on an initial feature image of a first stage, to obtain an intermediate feature image; and
performing merging processing on the intermediate feature image to obtain an output image,
wherein the cyclic scaling processing comprises hierarchically-nested scaling processing of N−1 stages, and scaling processing of each stage comprises down-sampling processing, concatenating processing, up-sampling processing, and residual link addition processing;
down-sampling processing of an i-th stage performs, based on an input of scaling processing of the i-th stage, down-sampling to obtain a down-sampling output of the i-th stage,
concatenating processing of the i-th stage performs, based on the down-sampling output of the i-th stage and an initial feature image of an (i+1)-th stage, concatenating to obtain a concatenating output of the i-th stage,
up-sampling processing of the i-th stage obtains an up-sampling output of the i-th stage based on the concatenating output of the i-th stage, and
residual link addition processing of the i-th stage performs residual link addition between the input of the scaling processing of the i-th stage and the up-sampling output of the i-th stage, to obtain an output of the scaling processing of the i-th stage, wherein i=1, 2, . . . , N−1; and
scaling processing of a (j+1)-th stage is nested between down-sampling processing of a j-th stage and concatenating processing of the j-th stage, and an output of the down-sampling processing of the j-th stage serves as an input of the scaling processing of the (j+1)-th stage, wherein j=1, 2, . . . , N−2,
wherein the obtaining, based on the input image, the initial feature images of the N stages with resolutions from high to low, comprises:
concatenating the input image with a random noise image to obtain a concatenating input image; and
performing analysis processing of N different stages on the concatenating input image, to obtain the initial feature images of the N stages with resolutions from high to low, respectively.

2. The image processing method according to claim 1 , wherein the concatenating processing of the i-th stage performing, based on the down-sampling output of the i-th stage and the initial feature image of the (i+1)-th stage, concatenating to obtain the concatenating output of the i-th stage, comprises:
taking the down-sampling output of the i-th stage as an input of scaling processing of the (i+1)-th stage, to obtain an output of the scaling processing of the (i+1)-th stage; and
concatenating the output of the scaling processing of the (i+1)-th stage with the initial feature image of the (i+1)-th stage to obtain the concatenating output of the i-th stage.

3. The image processing method according to claim 2 , wherein scaling processing of at least one stage is continuously performed a plurality of times, and an output of a former scaling processing serves as an input of a latter scaling processing.

4. The image processing method according to claim 3 , wherein the scaling processing of each stage is continuously performed twice.

5. The image processing method according to claim 1 , wherein among the initial feature images of the N stages, resolution of the initial feature image of the first stage is provided with a highest value, and the resolution of the initial feature image of the first stage is identical to resolution of the input image.

6. The image processing method according to claim 1 , wherein resolution of an initial feature image of a former stage is an integer multiple of resolution of an initial feature image of a latter stage.

7. The image processing method according to claim 1 , wherein obtaining the input image comprises:
obtaining an original input image with first resolution; and
performing resolution conversion processing on the original input image to obtain the input image with second resolution, wherein the second resolution is greater than the first resolution.

8. The image processing method according to claim 7 , wherein the resolution conversion processing is performed by using one selected from a group consisting of a bicubic interpolation algorithm, a bilinear interpolation algorithm, and a Lanczos interpolation algorithm.

9. The image processing method according to claim 1 , further comprising:
performing crop processing on the input image to obtain a plurality of sub-input images with an overlapping region;
obtaining, based on the input image, the initial feature images of the N stages with resolutions from high to low, comprises:
obtaining, based on each of the sub-input images, sub-initial feature images of N stages with resolutions from high to low, wherein N is a positive integer and N>2;

performing, based on the initial feature images of the second to N-th stages, the cyclic scaling processing on the initial feature image of the first stage to obtain the intermediate feature image, comprises:
performing, based on sub-initial feature images of second to N-th stages, cyclic scaling processing on a sub-initial feature image of a first stage, to obtain a sub-intermediate feature image; and

performing the merging processing on the intermediate feature image to obtain the output image, comprises:
performing merging processing on the sub-intermediate feature image to obtain a corresponding sub-output image, and
stitching sub-output images corresponding to the plurality of sub-input images into the output image.

10. The image processing method according to claim 9 , wherein the plurality of sub-input images are identical in size, centers of the plurality of sub-input images form a uniform and regular grid, an overlapping region of two adjacent sub-input images is provided with a constant size in both a row direction and a column direction, and a pixel value of each pixel point in the output image is expressed as:

Y
p

=

1

∑

k
=
1

T

s
k

⁢

∑

k
=
1

T

s
k

⁢

Y

k
,

(
p
)

,

wherein Y p represents a pixel value of any

Assignees

• Boe Technology Group Co Ltd

Inventors

• Navarrete Michelini Pablo
• Chen Wenbin
• Liu Hanwen
• Zhu Dan

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/094

  Adversarial learning · CPC title

• G06T3/4046 Primary

  using neural networks · CPC title