Training super-resolution convolutional neural network model using a high-definition training image, a low-definition training image, and a mask image

Patent metadata

Field	Value
Publication number	US-11704771-B2
Application number	US-201716759870-A
Country	US
Kind code	B2
Filing date	Dec 1, 2017
Priority date	Dec 1, 2017
Publication date	Jul 18, 2023
Grant date	Jul 18, 2023

Abstract

An image processing method and a device, where the image processing method is performed by a terminal having a digital zoom function, and the method includes determining a target zoom magnification based on a selection input of a user, collecting a to-be-processed image, and processing the to-be-processed image using a target super-resolution convolutional neural network model to obtain a processed image corresponding to the target zoom magnification, where the target super-resolution convolutional neural network model is obtained by training a super-resolution convolutional neural network model using a high-definition training image, a low-definition training image, and a mask image.

First claim

What is claimed is:

1. An image processing method implemented by a terminal, comprising:
training a super-resolution convolutional neural network model using a high-definition training image, a low-definition training image, and a mask image to obtain a first target super-resolution convolutional neural network model;
enabling a photographing function of the terminal;
enabling a zoom function of the terminal;
receiving a selection input of a user;
determining a target zoom magnification based on the selection input;
collecting a to-be-processed image;
processing the to-be-processed image using the first target super-resolution convolutional neural network model to obtain a processed image corresponding to the target zoom magnification by:
identifying that the target zoom magnification is greater than a maximum optical zoom magnification of the terminal; and
processing, in response to the identifying, the to-be-processed image using the first target super-resolution convolutional neural network model; and

displaying the processed image.

2. The image processing method of claim 1 , further comprising storing a second target super-resolution convolutional neural network model single magnification, and wherein processing the to-be-processed image further comprises:
determining whether the second target super-resolution convolutional neural network model single magnification is equal to the target zoom magnification; and
either:
processing the to-be-processed image using a second target super-resolution convolutional neural network model and outputting the processed image when the second target super-resolution convolutional neural network model single magnification is equal to the target zoom magnification; or
processing the to-be-processed image using a Y-magnification target super-resolution convolutional neural network model to obtain an intermediate result and performing (X-Y)x zoom on the intermediate result using a linear interpolation algorithm to output the processed image when the second target super-resolution convolutional neural network model single magnification does not equal to the target zoom magnification, wherein X is the target zoom magnification, and wherein Y is a maximum zoom magnification that is less than X and that is in the second target super-resolution convolutional neural network model single magnification.

3. The image processing method of claim 1 , further comprising storing a second target super-resolution convolutional neural network model comprising a plurality of magnifications, and wherein processing the to-be-processed image further comprises:
determining whether the magnifications comprise the target zoom magnification; and
either:
inputting the to-be-processed image into the second target super-resolution convolutional neural network model and processing the to-be-processed image using the second target super-resolution convolutional neural network model to output the processed image when the magnifications comprise the target zoom magnification; or
processing the to-be-processed image using the second target super-resolution convolutional neural network model to obtain an intermediate result and performing (X-Z)x zoom on the intermediate result using a linear interpolation algorithm to output the processed image when the magnifications do not comprise the target zoom magnification, wherein X is the target zoom magnification, and wherein Z is a maximum zoom magnification that is less than X and that is in the magnifications.

4. The image processing method of claim 1 , further comprising:
constructing, using an image registration algorithm, a training image pair of the high-definition training image and the low-definition training image that are photographed for a same scenario;
extracting an area of interest in the high-definition training image according to a preset rule;
assigning a first weight to the area of interest and a second weight to a second area in the high-definition training image to generate the mask image comprising a same size as the high-definition training image;
inputting the high-definition training image, the low-definition training image, and the mask image into the super-resolution convolutional neural network model;
calculating a loss cost result in each of the area of interest and the second area based on the first weight and the second weight; and
obtaining the first target super-resolution convolutional neural network model based on the loss cost result.

5. The image processing method of claim 4 , wherein obtaining the first target super-resolution convolutional neural network model based on the loss cost result further comprises:
determining whether the loss cost result meets a preset condition; and
either:
adjusting the super-resolution convolutional neural network model until an adjusted super-resolution convolutional neural network model meets the preset condition to obtain the first target super-resolution convolutional neural network model when the loss cost result does not meet the preset condition; or
setting the super-resolution convolutional neural network model as the first target super-resolution convolutional neural network model when the loss cost result meets the preset condition.

6. The image processing method of claim 4 , wherein extracting the area of interest in the high-definition training image further comprises:
extracting high-frequency information in the high-definition training image using a high-frequency extraction algorithm and setting an area in which the high-frequency information is located as the area of interest;
extracting face information in the high-definition training image using a face detection algorithm and setting the face information as the area of interest; or
extracting different objects as the area of interest using an image segmentation algorithm.

7. The image processing method of claim 4 , wherein extracting the area of interest in the high-definition training image further comprises:
extracting high-frequency information in the high-definition training image using a high-frequency extraction algorithm; and
setting an area in which the high-frequency information is located as the area of interest.

8. The image processing method of claim 4 , wherein extracting the area of interest in the high-definition training image further comprises extracting face information in the high-definition training image using a face detection algorithm and setting the face information as the area of interest.

9. The image processing method of claim 4 , wherein extracting the area of interest in the high-definition training image further comprises extracting different objects as the area of interest using an image segmentation algorithm.

10. The image processing method of claim 1 , further comprising:
receiving a photographing command from the user using a camera operation screen or a hardware photographing button; and
storing, in response to the photographing command, the processed image in a memory of the terminal.

11. An image processing method, comprising:
photographing a high-definition training image and a low-definition training image for a same scenario;
extracting an area of interest in the high-definition training image according to a preset rule;
assigning a first weight to the area of interest and a second weight to a second area in the high-definition training image to generate a mask image comprising a same size as the high-definition training image; and
training a super-resolution convolutional neural network model using the high-definition training image, the low-definition training image, and the mask image to generate a target super-resolu

Assignees

• Huawei Tech Co Ltd

Inventors

• Zhang Yunchao
• Chen Shuai
• Jia Zhiping
• Miao Lei

Classifications

• G06T3/4053 Primary

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title

• G06T3/4046 Primary

  using neural networks · CPC title

• H04N23/69

  Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming · CPC title

• H04N23/815

  for controlling the resolution by using a single image · CPC title