Image processing device for improving details of an image, and operation method of the same

Patent metadata

Field	Value
Publication number	US-11921822-B2
Application number	US-201917299607-A
Country	US
Kind code	B2
Filing date	Oct 16, 2019
Priority date	Dec 4, 2018
Publication date	Mar 5, 2024
Grant date	Mar 5, 2024

Abstract

Provided are an image processing apparatus and an operation method of the image processing apparatus. The image processing apparatus includes a memory storing one or more instructions, and a processor configured to execute the one or more instructions stored in the memory to, by using one or more convolution neural networks, extract target features by performing a convolution operation between features of target regions having same locations in a plurality of input images and a first kernel set, extract peripheral features by performing a convolution operation of features of peripheral regions located around the target regions in the plurality of input images and a second kernel set, and determine a feature of a region corresponding to the target regions in an output image, based on the target features and the peripheral features.

First claim

The invention claimed is:

1. An image processing apparatus comprising:
a memory storing one or more instructions; and
a processor configured to execute the one or more instructions stored in the memory,
wherein the processor is configured to, by using one or more convolution neural networks,
extract target features by performing a convolution operation between features of target regions having same locations in a plurality of input images and a first kernel set,
extract peripheral features by performing a convolution operation between features of peripheral regions located around the target regions in the plurality of input images and a second kernel set, and
determine a feature of a region in an output image, corresponding to the target regions, based on the target features and the peripheral features, and
wherein the plurality of input images comprise a first input image and a second input image,
wherein the target regions comprise a first target region and a second target region having same locations in the first input image and the second input image, respectively, and
wherein the peripheral regions comprise a first peripheral region located around the first target region in the first input image and a second peripheral region located around the second target region in the second input image.

2. The image processing apparatus of claim 1 , wherein the processor is further configured to
extract the first peripheral region in a range where a distance from the first target region is within a preset distance, from the first input image, and
extract the second peripheral region in a range where a distance from the second target region is within the preset distance, from the second input image.

3. The image processing apparatus of claim 1 , wherein the processor is further configured to
determine a first feature similarity between each of a plurality of regions included in the first input image and the first target region, and determine the first peripheral region based on the first feature similarity, and
determine a second feature similarity between each of a plurality of regions included in the second input image and the second target region, and determine the second peripheral region based on the second feature similarity.

4. The image processing apparatus of claim 3 , wherein the first peripheral region is a region having a feature most similar to a feature of the first target region from among the plurality of regions included in the first input image, and
the second peripheral region is a region having a feature most similar to a feature of the second target region from among the plurality of regions included in the second input image.

5. The image processing apparatus of claim 1 , wherein the processor is further configured to
determine a first weight that is applied to the first peripheral region, based on a distance between the first target region and the first peripheral region,
determine a second weight that is applied to the second peripheral region, based on a distance between the second target region and the second peripheral region, and
extract the peripheral features by applying the first weight and the second weight.

6. The image processing apparatus of claim 5 , wherein the processor is further configured to determine the first weight as a larger value as the first target region and the first peripheral region are closer to each other, and determine the second weight as the larger value as the second target region and the second peripheral region are closer to each other.

7. The image processing apparatus of claim 1 , wherein the processor is further configured to
determine a first weight that is applied to the first peripheral region, based on a first similarity between a feature of the first target region and a feature of the first peripheral region,
determine a second weight that is applied to the second peripheral region, based on a second similarity between a feature of the second target region and a feature of the second peripheral region, and
extract the peripheral features by applying the first weight and the second weight.

8. The image processing apparatus of claim 7 , wherein the processor is further configured to determine the first weight as a larger value as the first similarity is larger, and determine the second weight as the larger value as the second similarity is larger.

9. An operation method of an image processing apparatus, the operation method comprising:
extracting target features by performing a convolution operation between features of target regions having same locations in a plurality of input images and a first kernel set;
extracting peripheral features by performing a convolution operation between features of peripheral regions located around the target regions in the plurality of input images and a second kernel set; and
determining a feature of a region in an output image, corresponding to the target regions, based on the target features and the peripheral features, and
wherein the plurality of input images comprise a first input image and a second input image,
wherein the target regions comprise a first target region and a second target region having same locations in the first input image and the second input image, respectively, and
wherein the peripheral regions comprise a first peripheral region located around the first target region in the first input image and a second peripheral region located around the second target region in the second input image.

10. The operation method of claim 9 , wherein the extracting of the peripheral features comprises:
extracting the first peripheral region in a range where a distance from the first target region is within a preset distance, from the first input image; and
extracting the second peripheral region in a range where a distance from the second target region is within the preset distance, from the second input image.

11. The operation method of claim 9 , wherein the extracting of the peripheral features comprises:
determining a first feature similarity between each of a plurality of regions included in the first input image and the first target region, and determining the first peripheral region based on the first feature similarity; and
determining a second feature similarity between each of a plurality of regions included in the second input image and the second target region, and determining the second peripheral region based on the second feature similarity.

12. The operation method of claim 11 , wherein the determining of the first peripheral region comprises determining, as the first peripheral region, a region having a feature most similar to a feature of the first target region from among the plurality of regions included in the first input image, and
the determining of the second peripheral region comprises determining, as the second peripheral region, a region having a feature most similar to a feature of the second target region from among the plurality of regions included in the second input image.

13. The operation method of claim 9 , wherein the extracting of the peripheral features comprises:
determining a first weight that is applied to the first peripheral region, based on a distance between the first target region and the first peripheral region;
determining a second weight that is applied to the second peripheral region, based on a distance between the second target region and the second peripheral region; and
extracting the peripheral features by applying the first weight and the second weight.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Park Jaeyeon
• Ahn Iljun
• Park Yongsup
• Kwak Jaehee
• Lee Tammy
• Cheon Minsu

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06F18/22 Primary

  Matching criteria, e.g. proximity measures · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title

• G06T7/11 Primary

  Region-based segmentation · CPC title

• G06V10/40

  Extraction of image or video features · CPC title