Image processing device, method, and program

Patent metadata

Field	Value
Publication number	US-12039736-B2
Application number	US-201917413429-A
Country	US
Kind code	B2
Filing date	Dec 9, 2019
Priority date	Dec 13, 2018
Publication date	Jul 16, 2024
Grant date	Jul 16, 2024

Abstract

Labels can be accurately identified even for an image with a resolution not used in training data. Based on an input image, a resolution of the input image, and a resolution of a training image used for training a trained model of assigning labels to pixels of an image, a plurality of low-resolution images are generated from the input image by using a plurality of shift amounts for a pixel correspondence between the input image and the respective low-resolution images with a resolution corresponding to the training image, the low-resolution images are input to the trained model, a plurality of low-resolution label images is output in which pixels of the respective low-resolution images are assigned labels, and a label image is output in which labels for pixels of the input image are obtained, based on the shift amounts used for generating the low-resolution images and the low-resolution label images.

First claim

The invention claimed is:

1. An image processing device comprising:
a downsampler configured to generate, based on an input image, a resolution of the input image, and a resolution of a training image used for training a trained model of assigning labels to pixels of an image, a plurality of low-resolution images from the input image by using a plurality of shift amounts for a pixel correspondence between the input image and the respective low-resolution images with a resolution corresponding to the training image, and
output the generated low-resolution images and the shift amounts used for generating the low-resolution images;

a semantic segmentation processor configured to input the low-resolution images to the trained model, and
output a plurality of low-resolution label images in which labels are respectively assigned to pixels of the respective low-resolution images; and

an upsampler configured to output, based on the shift amounts used for generating the low-resolution images and the low-resolution label images, a label image in which labels for pixels of the input image are obtained,
wherein the upsampler outputs the label image:
by calculating, for each of the low-resolution label images, a separate label image in which a score of the label that is based on an overlapping ratio of the pixel of the low-resolution label image on the pixel of the input image is assigned to each pixel of the input image, based on a result of allocating the low-resolution label image to the input image according to the corresponding shift amount, and
by integrating, for each pixel, the scores of the labels assigned to the corresponding pixels of the calculated separate label images to determine a label for the each pixel.

2. The image processing device according to claim 1 , wherein the resolution of the input image is higher than the resolution of the training image.

3. The image processing device according to claim 1 , wherein at least one of the plurality of low-resolution label images includes a label associated with a pixel in the at least one of the plurality of low-resolution label images.

4. The image processing device according to claim 1 , wherein the resolution of the input image is higher than the resolution of the training image.

5. The image processing device according to claim 1 , wherein at least one of the plurality of low-resolution label images includes a label associated with a pixel in the at least one of the plurality of low-resolution label images.

6. An image processing method, the method comprising:
generating, by a downsampler, based on an input image, a resolution of the input image, and a resolution of a training image used for training a trained model of assigning labels to pixels of an image, a plurality of low-resolution images from the input image by using a plurality of shift amounts for a pixel correspondence between the input image and the respective low-resolution images with a resolution corresponding to the training image, and outputting the generated low-resolution images and the shift amounts used for generating the low-resolution images;
receiving, by a semantic segmentation processor, the low-resolution images to the trained model;
outputting, by the semantic segmentation processor, a plurality of low-resolution label images in which labels are respectively assigned to pixels of the respective low-resolution images; and
outputting, by an upsampler, based on the shift amounts used for generating the low-resolution images and the low-resolution label images, a label image in which labels for pixels of the input image are obtained,
wherein the upsampler outputs the label image:
by calculating, for each of the low-resolution label images, a separate label image in which a score of the label that is based on an overlapping ratio of the pixel of the low-resolution label image on the pixel of the input image is assigned to each pixel of the input image, based on a result of allocating the low-resolution label image to the input image according to the corresponding shift amount, and
by integrating, for each pixel, the scores of the labels assigned to the corresponding pixels of the calculated separate label images to determine a label for the each pixel.

7. The image processing method according to claim 6 , wherein the resolution of the input image is higher than the resolution of the training image.

8. The image processing method according to claim 6 , wherein at least one of the plurality of low-resolution label images includes a label associated with a pixel in the at least one of the plurality of low-resolution label images.

9. The image processing method according to claim 6 , wherein the resolution of the input image is higher than the resolution of the training image.

10. The image processing method according to claim 6 , wherein at least one of the plurality of low-resolution label images includes a label associated with a pixel in the at least one of the plurality of low-resolution label images.

11. A computer-readable non-transitory recording medium storing a computer-executable program instructions that when executed by a processor cause a computer system to:
generate, by a downsampler, based on an input image, a resolution of the input image, and a resolution of a training image used for training a trained model of assigning labels to pixels of an image, a plurality of low-resolution images from the input image by using a plurality of shift amounts for a pixel correspondence between the input image and the respective low-resolution images with a resolution corresponding to the training image, and outputting the generated low-resolution images and the shift amounts used for generating the low-resolution images;
receive, by a semantic segmentation processor, the low-resolution images to the trained model;
output, by the semantic segmentation processor, a plurality of low-resolution label images in which labels are respectively assigned to pixels of the respective low-resolution images; and
output, by an upsampler, based on the shift amounts used for generating the low-resolution images and the low-resolution label images, a label image in which labels for pixels of the input image are obtained,
wherein the upsampler outputs the label image:
by calculating, for each of the low-resolution label images, a separate label image in which a score of the label that is based on an overlapping ratio of the pixel of the low-resolution label image on the pixel of the input image is assigned to each pixel of the input image, based on a result of allocating the low-resolution label image to the input image according to the corresponding shift amount, and
by integrating, for each pixel, the scores of the labels assigned to the corresponding pixels of the calculated separate label images to determine a label for the each pixel.

12. The computer-readable non-transitory recording medium according to claim 11 , wherein the resolution of the input image is higher than the resolution of the training image.

13. The computer-readable non-transitory recording medium according to claim 11 , wherein at least one of the plurality of low-resolution label images includes a label associated with a pixel in the at least one of the plurality of low-resolution label images.

14. The computer-readable non-transitory recording medium according to claim 11 , wherein the resolution of the input image is higher than the resolution of the training image.

15. The computer-readable non-transitory recording medium according to claim 11 , wherein at least one of the plurality of low-resolution label images inc

Assignees

• Nippon Telegraph & Telephone

Inventors

• Tsukatani Shunsuke
• Murasaki Kazuhiko
• Ando Shingo
• Sagata Atsushi

Classifications

• G06F18/214

  Generating training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title

• G06T2207/20081

  Training; Learning · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/774

  Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting · CPC title