Image processing method, image processing apparatus, image processing system, and learnt model manufacturing method

Patent metadata

Field	Value
Publication number	US-2020311981-A1
Application number	US-202016826370-A
Country	US
Kind code	A1
Filing date	Mar 23, 2020
Priority date	Mar 29, 2019
Publication date	Oct 1, 2020
Grant date	—

Abstract

An image processing method comprising steps of obtaining a first map representing a region outside a dynamic range of an input image based on a signal value in the input image and a threshold of the signal value, and inputting input data including the input image and the first map and executing a recognition task or a regression task.

First claim

What is claimed is:

1 . An image processing method comprising steps of:
obtaining a first map representing a region outside a dynamic range of an input image based on a signal value in the input image and a threshold of the signal value; and inputting input data including the input image and the first map and executing a recognition task or a regression task.

2 . The image processing method according to claim 1 , wherein the threshold of the signal value is set based on at least one of a luminance saturation value and a black level in the input image.

3 . The image processing method according to claim 1 , wherein the first map is a map representing at least one of a luminance saturated area and a blocked-up shadow area in the input image.

4 . The image processing method according to claim 1 , wherein the input data includes the input image and the first map as a channel component.

5 . The image processing method according to claim 4 , wherein the inputting step inputs only one of the input image and the first map to a first layer of the neural network, concatenates, in a channel direction, a feature map that is an output from at least the first layer, with the other of the input image and the first map that has not been input to the first layer, and inputs concatenated data to a subsequent layer of the neural network.

6 . The image processing method according to claim 4 , wherein the inputting step branches an input part of the neural network, converts the input image and the first map into feature maps in different layers, concatenates the feature maps, and inputs that to a subsequent layer.

7 . The image processing method according to claim 1 , wherein pixel numbers per one channel are equal to each other between the input image and the first map.

8 . The image processing method according to claim 1 , wherein the task is to deblur the input image.

9 . The image processing method according to claim 1 , further comprising steps of:
obtaining a weight map based on the signal value in the input image and the threshold of the signal value; and generating a weighted average image based on the output from the neural network, the input image, and the weight map.

10 . The image processing method according to claim 9 , wherein the input image includes a plurality of color components, and
wherein in the input image, when luminance saturation or a blocked-up shadow occurs in all of a target pixel and pixels having a color component different from that of the target pixel in a predetermined area, the weight map is generated so that a weight at a position of the target pixel in the input image is larger than the output from the neural network.

11 . The image processing method according to claim 9 , wherein the input image has a plurality of color components, and
wherein in the input image, when neither luminance saturation nor a blocked-up shadow occurs in any of the target pixel and a pixel having a color component different from that of the target pixel in the predetermined area, the weight map is generated so that a weight at the position of the target pixel in the input image is smaller than the output from the neural network.

12 . An image processing apparatus comprising:
an obtaining unit configured to obtain a first map representing a region outside a dynamic range of an input image based on a signal value in the input image and a threshold of the signal value; and a processing unit configured to input data including the input image and the first map to a neural network, and to execute a recognition task or a regression task, wherein at least one processor or circuit is configured to perform a function of at least one of the units.

13 . The image processing apparatus according to claim 12 , further comprising a memory configured to store weight information used in the neural network.

14 . A non-transitory computer-readable storage medium storing a computer program that causes a computer to execute the image processing method according to claim 1 .

15 . An image processing system comprising:
a first apparatus; and a second apparatus communicable with the first apparatus, wherein the first apparatus includes a transmitter configured to transmit a request to make the second apparatus execute processing on a captured image, wherein the second apparatus includes: a receiver configured to receive the request transmitted by the transmitter; an obtainer configured to obtain a first map representing a region outside a dynamic range of the captured image based on a signal value in the captured image and a threshold of the signal value; a processor configured to input data including the captured image and the first map to a neural network and to execute a recognition task or a regression task; and a transmitter configured to transmit a result of the task.

16 . An image processing method comprising steps of:
obtaining a training image, a first map representing a region outside a dynamic range of the training image based on a signal value in the training image and a threshold of the signal value, and ground truth data; and making a neural network learn for executing a recognition task or a regression task, using input data including the training image and the first map, and the ground truth data.

17 . A non-transitory computer-readable storage medium storing a computer program that causes a computer to execute the image processing method according to claim 16 .

18 . A learnt model manufacturing method comprising steps of:
obtaining a training image, a first map representing a region outside a dynamic range of the training image based on a signal value in the training image and a threshold of the signal value, and ground truth data; and making a neural network learn for executing a recognition task or a regression task using input data including the training image and the first map, and the ground truth data.

19 . An image processing apparatus comprising:
an obtaining unit configured to obtain a training image, a first map representing a region outside a dynamic range of the training image based on a signal value in the training image and a threshold of the signal value, and ground truth data; and a learning unit configured to make a neural network learn for executing a recognition task or a regression task using input data including the training image and the first map, and the ground truth data, wherein at least one processor or circuit is configured to perform a function of at least one of the units.

Assignees

• Canon Kk

Inventors

• Hiasa Norihito

Classifications

• G06F2218/08

  Feature extraction · CPC title

• G06F18/00 Primary

  Pattern recognition · CPC title

• H04N23/741

  by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors · CPC title

• H04N25/57

  Control of the dynamic range · CPC title

• G06N3/09

  Supervised learning · CPC title