Image processing method, image processing apparatus, electronic device and computer-readable storage medium

Patent metadata

Field	Value
Publication number	US-12400302-B2
Application number	US-202217582211-A
Country	US
Kind code	B2
Filing date	Jan 24, 2022
Priority date	Jan 12, 2021
Publication date	Aug 26, 2025
Grant date	Aug 26, 2025

Abstract

An image processing method, an image processing apparatus, an electronic device, and a computer-readable storage medium, relating to the technical field of image processing are provided. The image processing method may include performing blur classification on pixels of an image to obtain a classification mask image; and determining a blurred area of the image based on the classification mask image.

First claim

What is claimed is:

1. An image processing method, comprising:
performing blur classification on pixels of an image by:
applying an adaptive portioning method, in which the image is divided into image blocks and any image block with a confidence score within a predetermined confidence range is recursively subdivided into smaller image blocks immediately before further image portioning causes the confidence score of any image block to fall outside the present conference range, and
processing the image blocks, which has varying sizes as a result of the adaptive portioning method, using a graph convolutional network;

determining a blurred area and a clear area in the image based on the blur classification;
determining a mask matrix from a classification mask image that is obtained as a result of the blur classification;
obtaining a masked similarity matrix by applying the mask matrix that indicates clear pixels in the clear area as active elements to be considered and blurred pixels in the blurred area to be ignored, to a similarity matrix that indicate similarity values between the blurred pixels and the clear pixels, wherein the mask matrix, a positive relationship of the clear pixels to the blurred pixels is indicated to enhance the blurred pixels using the clear pixels, and a negative relationship of the blurred pixels to the clear pixels is indicated to prevent the blurred pixels from damaging the clear pixels; and
deblurring the image based on the masked similarity matrix.

2. The method according to claim 1 , wherein the performing the blur classification comprises:
performing feature extraction on the image via a plurality of cascaded first feature extraction layers of a feature extraction neural network to obtain at least one first feature image;
performing feature extraction on the at least one first feature image via a second feature extraction layer of the feature extraction neural network, based on a relationship between different pixels in the first feature image, to obtain a second feature image; and
generating the classification mask image based on the second feature image.

3. The method according to claim 2 , wherein the performing the feature extraction on the at least one first feature image comprises:
dividing the at least one first feature image into the image blocks;
extracting, by the graph convolutional network, local features of each of the image blocks and global features between adjacent image blocks;
fusing the local features and the global features to obtain a second fused feature; and
generating the second feature image based on the second fused feature.

4. The method according to claim 3 , wherein at least one of the image blocks obtained at a first partitioning step of the adaptive portioning method comprises both blurred pixels and clear pixels, and none of the image blocks obtained after a last partitioning step of the adaptive portioning method comprises both blurred pixels and clear pixels.

5. The method according to claim 3 , wherein the extracting the global features comprises:
performing dimension reduction on each of the image blocks to obtain dimension reduced image blocks; and
extracting from the dimension reduced image blocks the global features between the adjacent image blocks.

6. The method according to claim 1 , wherein the deblurring the image comprises:
obtaining a first recovery image based on a determination of the blurred area of the image;
extracting the clear pixels in clear areas in the image based on the classification mask image that is obtained as a result of the blur classification; and
replacing pixels corresponding to clear areas in the first recovery image with the clear pixels to obtain a second recovery image.

7. The method according to claim 6 , wherein the deblurring the image further comprises:
performing feature extraction on the image to obtain a first extracted feature; and
performing recovery, by at least one recovery neural network and based on the first extracted feature, to obtain the first recovery image.

8. The method according to claim 7 , wherein the performing the recovery comprises:
selecting a scale from a plurality of preset scales based on an input feature of the at least one recovery neural network;
based on the input feature, performing the feature extraction according to the selected scale to obtain a scale feature; and
determining and outputting a recovery feature based on the scale feature.

9. The method according to claim 8 , wherein the determining and outputting the recovery feature comprises:
selecting a channel from a plurality of channels based on the scale feature; and
determining and outputting the recovery feature based on a channel feature corresponding to the selected channel.

10. The method according to claim 7 , wherein the performing the recovery comprises:
performing dimension raising on a recovery feature output from the least one recovery neural network to obtain the first recovery image.

11. The method according to claim 1 ,
wherein a value in the mask matrix represents a relationship between a first pixel and a second pixel in the classification mask image, the value in the mask matrix is 0 when the first pixel is a clear pixel and the second pixel is a blurred pixel, and the value in the mask matrix is 1 when the first pixel is the blurred pixel and the second pixel is the clear pixel.

12. The method according to claim 6 , wherein the deblurring the image further comprises:
obtaining a twin image of the image;
performing feature extraction on the image and the twin image based on a twin network to obtain a first extracted feature and a second extracted feature;
aligning the first extracted feature and the second extracted feature to obtain an aligned feature; and
deblurring the image based on the aligned feature to obtain the first recovery image.

13. The method according to claim 12 , wherein the aligning the first extracted feature and the second extracted feature comprises:
setting any one of the first extracted feature and the second extracted feature as a query feature and the other one as a key-value feature; and
fusing the query feature into the key-value feature to obtain the aligned feature.

14. An image processing apparatus, comprising:
a memory storing one or more instructions; and
at least one processor configured to execute the one or more instructions to:
perform blur classification on pixels of an image by:
applying an adaptive portioning method, in which the image is divided into image blocks and any image block with a confidence score within a predetermined confidence range is recursively subdivided into smaller image blocks immediately before further image portioning causes the confidence score of any image block to fall outside the present conference range, and
processing the image blocks, which has varying sizes as a result of the adaptive portioning method, using a graph convolutional network;

determine a blurred area and a clear area in the image based on the blur classification;
obtain a masked similarity matrix by applying a mask matrix that indicates clear pixels in the clear area as active elements to be considered and blurred pixels in the blurred area to be ignored, to a similarity matrix that indicate similarity values between the blurred pixels and the clear pixels, wherein the mask matrix, a positive relationship of the clear pixels to the blurred pixels is indicated to enhance the blurred pixels using the clear pixels, and a negative relationship of the blurred pixels to the clear pixels is indicated to prevent the blurred pixels from damagin

Assignees

• Samsung Electronics Co Ltd

Inventors

• Zhang Dafeng
• Gao Yanjun
• Dai Bin
• Jin Zhezhu
• Wen Wei

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06T5/70

  Denoising; Smoothing · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/806

  of extracted features · CPC title