Method and image processing device for image super resolution, image enhancement, and convolutional neural network model training

Patent metadata

Field	Value
Publication number	US-11301965-B2
Application number	US-201916708300-A
Country	US
Kind code	B2
Filing date	Dec 9, 2019
Priority date	Sep 3, 2019
Publication date	Apr 12, 2022
Grant date	Apr 12, 2022

Abstract

The disclosure provides methods and image processing devices for image super resolution, image enhancement, and convolutional neural network (CNN) model training. The method for image super resolution includes the following steps. An original image is received, and a feature map is extracted from the original image. The original image is segmented into original patches. Each of the original patches is classified respectively into one of patch clusters according to the feature map. The original patches are processed respectively by different pre-trained CNN models according to the belonging patch clusters to obtain predicted patches. A predicted image is generated based on the predicted patches.

First claim

What is claimed is:

1. A method for image super resolution, comprising:
receiving an original image;
extracting a feature map from the original image;
segmenting the original image into a plurality of original patches;
classifying each of the original patches respectively into one of a plurality of patch clusters according to the feature map;
processing the original patches respectively by a plurality of different pre-trained convolutional neural network (CNN) models according to the belonging patch clusters to obtain a plurality of predicted patches; and
generating a predicted image based on the predicted patches.

2. The method according to claim 1 , wherein the step of extracting the feature map from the original image comprises:
applying a feature extractor on the original image, wherein each pixel in the original image has a feature value at a same coordinate in the feature map.

3. The method according to claim 2 , wherein the feature value of a center pixel of a patch among the original patches represents the feature value of the patch.

4. The method according to claim 2 , wherein the step of classifying each of the original patches respectively into one of the patch clusters according to the feature map comprises:
classifying original patches with the same feature value into one patch cluster, wherein each of the clusters corresponds to a different one of the pre-trained CNN models.

5. The method according to claim 2 , wherein the feature extractor is a binary adaptive dynamic range coder (ADRC).

6. The method according to claim 2 , wherein the feature extractor is edge detector.

7. The method according to claim 1 , wherein the step of processing the original patches respectively by the different pre-trained CNN models comprises:
processing the plurality of original patches respectively by the corresponding pre-trained CNN models to obtain the predicted patches.

8. The method according to claim 1 , wherein each of the pre-trained CNN models comprises a plurality of convolution layers.

9. The method according to claim 1 , wherein the step of processing the original patches respectively by the different pre-trained CNN models according to the belonging patch clusters comprising:
upscaling the original patches to a target resolution; and
processing the upscaled original patches into the pre-trained CNN models according to which cluster they belong to.

10. The method according to claim 1 , wherein each of the pre-trained CNN models comprises a plurality of convolution layers and an upscaling layer, and wherein the upscaling layer in each of the pre-trained CNN models upscales an output of one of the convolution layers to a target resolution.

11. The method according to claim 1 , wherein the step of generating the predicted image based on the predicted patches comprises:
assembling the predicted patches into the predicted image.

12. The method according to claim 1 , wherein a resolution of the original image is lower than a resolution of the predicted image.

13. The method according to claim 1 , wherein a resolution of each of the original patches is lower than a resolution of each of the predicted patches.

14. A method for convolutional neural network (CNN) model training, comprising:
receiving a plurality of high-resolution training images;
down-sampling the high-resolution training images to generate a plurality of low-resolution training images respectively corresponding to the high-resolution training images;
extracting feature maps from the low-resolution training image;
segmenting each of the low-resolution training images and the high-resolution training images respectively into a plurality of low-resolution training patches and a plurality of high-resolution training patches; and
classifying each of the low-resolution and high-resolution patch pairs respectively into one of a plurality of patch clusters according to the feature maps; and
learning a mapping function of each of a plurality of CNN models by using all of the low-resolution and high-resolution patch pairs of the patch clusters to generate a corresponding pre-trained CNN model.

15. A method for image enhancement, comprising:
receiving an original image;
extracting a feature map from the original image;
segmenting the original image into a plurality of original patches;
processing the original patches respectively by a plurality of different pre-trained convolutional neural network (CNN) models based on the feature map corresponding to the original patches to obtain a plurality of enhanced patches; and
generating an enhanced image based on the enhanced patches.

16. A method for convolutional neural network (CNN) model training, comprising:
receiving a plurality of high-quality training images;
downgrading the high-quality training images to generate a plurality of low-quality training image respectively corresponding to the high-quality training images;
extracting feature maps from the low-quality training images;
segmenting each of the low-quality training images and the high-quality training images respectively into a plurality of low-quality training patches and a plurality of high-quality training patches; and
learning a mapping function of each of a plurality of CNN models by using the feature maps, the low-quality training patches, and the high-quality training patches to generate a corresponding pre-trained CNN model.

17. An image processing device comprising:
a memory circuit, configured to store data and a plurality of different pre-trained convolutional neural network (CNN) model; and
a processing circuit, configured to:
receive an original image;
extract a feature map from the original image;
segment the original image into a plurality of original patches;
classify each of the original patches respectively into one of a plurality of patch clusters according to the feature map;
process the original patches respectively by the different pre-trained CNN models according to the belonging patch clusters to obtain a plurality of predicted patches; and
generate the predicted image based on the predicted patches.

18. An image processing device comprising:
a memory circuit, configured to store data; and
a processing circuit, configured to:
receive a plurality of high-resolution training images;
down-sample the high-resolution training images to generate a plurality of low-resolution training images respectively corresponding to the high-resolution training images;
extract feature maps from the low-resolution training image;
segment each of the low-resolution training images and the high-resolution training images respectively into a plurality of low-resolution training patches and a plurality of high-resolution training patches;
classify each of the low-resolution and high-resolution patch pairs respectively into one of a plurality of patch clusters according to the feature maps; and
learn a mapping function of each of a plurality of CNN models by using all of the low-resolution and high-resolution patch pairs of the patch clusters to generate a corresponding pre-trained CNN model.

19. An image processing device comprising:
a memory circuit, configured to store data and a plurality of different pre-trained convolutional neural network (CNN) model; and
a processing circuit, configured to:
receive an original image;
extract a feature map from the original image;
segment the original image into a plurality of original patches;
process the original patches respectively by a plurality of differ

Assignees

• Novatek Microelectronics Corp

Inventors

• Bai Yu

Classifications

• G06T3/4053 Primary

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G06F18/2148

  characterised by the process organisation or structure, e.g. boosting cascade · CPC title

• G06N3/045

  Combinations of networks · CPC title