Image reconstruction method and device, apparatus, and non-transitory computer-readable storage medium

Patent metadata

Field	Value
Publication number	US-11810265-B2
Application number	US-201917299557-A
Country	US
Kind code	B2
Filing date	Dec 20, 2019
Priority date	Dec 20, 2018
Publication date	Nov 7, 2023
Grant date	Nov 7, 2023

Abstract

An image reconstruction method, device and apparatus and non-transitory computer-readable storage medium are disclosed. The method may include: determining norms of convolution kernels of each convolutional layer of a deep neural network model; determining the convolution kernels with norms greater than or equal to a preset threshold in each convolutional layer to obtain a target convolution kernel set of each convolutional layer; processing an input image of each convolutional layer by using the convolution kernels in the target convolution kernel set of each convolutional layer respectively, to obtain a first image processing result; obtaining a second image processing result by performing interpolation on an initial image; and determining a fusion result according to the first image processing result and the second image processing result and reconstructing the initial image according to the fusion result.

First claim

The invention claimed is:

1. An image reconstruction method, comprising:
determining norms of convolution kernels of each convolutional layer of a deep neural network model;
determining the convolution kernels with norms greater than or equal to a preset threshold in each convolutional layer to obtain a target convolution kernel set of each convolutional layer, wherein the target convolution kernel set of each convolutional layer comprises the convolution kernels with norms greater than or equal to the preset threshold;
processing an input image of each convolutional layer by using the convolution kernels in the target convolution kernel set of each convolutional layer respectively, to obtain a first image processing result for the deep neural network model;
obtaining a second image processing result by performing interpolation on an initial image; and
determining a fusion result according to the first image processing result and the second image processing result and reconstructing the initial image according to the fusion result;
wherein processing an input image of each convolutional layer by using the convolution kernels in the target convolution kernel set of each convolutional layer respectively comprises:
processing the input image by using the convolution kernels with norms greater than or equal to the preset threshold in response to a difference between a maximum norm and a minimum norm of the convolution kernels in the target convolution kernel set of each convolutional layer respectively being within a preset range; and
sorting the convolution kernels in a descending order of the norms in response to the difference between the maximum norm and minimum norm of the convolution kernels in the target convolution kernel set of each convolutional layer respectively not being within the preset range, and processing the input image by using first M convolution kernels in a sorting result, wherein M is a natural number greater than or equal to 1.

2. The image reconstruction method of claim 1 , wherein the deep neural network model comprises N convolutional layers, and determining the convolution kernels with norms greater than or equal to a preset threshold in each convolutional layer to obtain a target convolution kernel set of each convolutional layer comprises:
comparing the norms of the convolution kernels of an i th convolutional layer among the N convolutional layers with a preset threshold to determine the convolution kernels with norms greater than or equal to the preset threshold in the i th convolutional layer to obtain a first target convolution kernel subset of the i th convolutional layer; and
comparing the norms of the convolution kernels of an (i+1)th convolutional layer among the N convolutional layers with a preset threshold to determine the convolution kernels with norms greater than or equal to the preset threshold in the (i+1) th convolutional layer to obtain a second target convolution kernel subset of the (i+1) th convolutional layer;
wherein i is a natural number greater than 1 and less than N, and the (i+1) th convolutional layer represents a next convolutional layer to the i th convolutional layer.

3. The image reconstruction method of claim 2 , wherein processing an input image of each convolutional layer by using the convolution kernels in the target convolution kernel set of each convolutional layer respectively, to obtain a first image processing result for the deep neural network model comprises:
processing an input image of the i th convolutional layer by using the first target convolution kernel subset to obtain an image processing result of the i th convolutional layer;
taking the image processing result of the i th convolutional layer as an input image of the (i+1) th convolutional layer;
processing the input image of the (i+1) th convolutional layer by using the second target convolution kernel subset to obtain an image processing result of the (i+1) th convolutional layer; and
taking the image processing result of the N th convolutional layer as the first image processing result, when i+1=N.

4. The image reconstruction method of claim 1 , wherein determining a fusion result according to the first image processing result and the second image processing result comprises:
determining weights corresponding to the first image processing result and the second image processing result, respectively; and
determining a fusion result according to the first image processing result, the second image processing result, the weight corresponding to the first image processing result and the weight corresponding to the second image processing result.

5. The image reconstruction method of claim 4 , wherein determining weights corresponding to the first image processing result and the second image processing result respectively comprises:
determining the weight corresponding to the first image processing result according to a reliability coefficient of each convolutional layer; and
determining the weight corresponding to the second image processing result according to the weight corresponding to the first image processing result.

6. The image reconstruction method of claim 5 , further comprising:
determining the reliability coefficient of each convolutional layer according to a formula of:

t
i

=

1
-

∑

j
=

0
⁢

:

⁢
Q

-
Ω

⁢



f
ij

⁡

(
x
)



∑

j
=

0
⁢

:

⁢
Q

⁢



f
ij

⁡

(
x
)



where t i represents a reliability coefficient of the i th convolutional layer, |f ij (x)| represents the norm of the j th convolution kernel of the i th convolutional layer, Ω represents a set of the convolution kernels with norms greater

Assignees

• Sanechips Tech Co Ltd

Inventors

• Kong Dehui
• Xu Ke
• Zhang Xiao
• Han Bin
• Han Zhou
• Wang Hong
• Lu Guoning
• Huang Long
• Luo Sheng

Classifications

• G06T3/4007 Primary

  based on interpolation, e.g. bilinear interpolation (image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403) · CPC title

• G06F18/251

  of input or preprocessed data · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title