Image processing apparatus and operating method thereof

Patent metadata

Field	Value
Publication number	US-12573191-B2
Application number	US-202318136614-A
Country	US
Kind code	B2
Filing date	Apr 19, 2023
Priority date	May 9, 2022
Publication date	Mar 10, 2026
Grant date	Mar 10, 2026

Abstract

An image processing apparatus for processing an image by using one neural network, includes: a memory storing one instruction; and one processor configured to execute the one instruction to: obtain first feature data, based on a first image, obtain pieces of second feature data corresponding to first areas of the first image by performing first image processing on the first feature data, the first areas comprising a first number of pixels, obtain third feature data, based on the first image, obtain pieces of fourth feature data corresponding to second areas of the first image, by performing second image processing on the third feature data, the second areas comprising a second number of pixels that is greater than the first number, and generate a second image, based on the pieces of second feature data and the pieces of fourth feature data.

First claim

What is claimed is:

1 . An image processing apparatus for processing an image by using at least one neural network, the image processing apparatus comprising:
a memory storing at least one instruction; and at least one processor configured to execute the at least one instruction to:
obtain first feature data, based on a first image,
divide the first feature data into a plurality of first patches, each of the plurality of first patches comprising M pixels,
obtain pieces of second feature data corresponding to each of pixels included in each of the plurality of first patches by performing a first self-attention operation on the pixels included in the same patch for each of the plurality of first patches,
obtain third feature data, based on the first image,
divide the pieces of second feature data into a plurality of second patches, each of the plurality of second patches comprising M pixels,
obtain pieces of fourth feature data corresponding to each of the plurality of second patches, by performing a second self-attention operation on the plurality of second patches, and
generate a second image, based on the pieces of second feature data and the pieces of fourth feature data.

2 . The image processing apparatus of claim 1 , wherein the at least one processor is further configured to execute the at least one instruction to:
obtain the pieces of second feature data respectively corresponding to the plurality of first patches, based on information about peripheral areas of each of the plurality of first patches, and obtain the pieces of fourth feature data respectively corresponding to the plurality of second patches, based on information about peripheral areas of each of the plurality of second patches.

3 . The image processing apparatus of claim 1 , wherein each of the plurality of first patches comprises one pixel.

4 . The image processing apparatus of claim 1 , wherein the at least one processor is further configured to execute the at least one instruction to:
obtain pieces of query data, pieces of key data, and pieces of value data based on the first feature data, the pieces of query data, the pieces of key data, and the pieces of value data respectively corresponding to the plurality of first patches, obtain a weight matrix, based on the pieces of query data and the pieces of key data, and obtain the pieces of second feature data, based on the pieces of value data and the weight matrix.

5 . The image processing apparatus of claim 4 , wherein the at least one processor is further configured to execute the at least one instruction to:
obtain a correlation matrix, based on the pieces of query data and the pieces of key data, and obtain the weight matrix by applying, to the correlation matrix, a position bias based on a size of the first image and sizes of images used to train the at least one neural network.

6 . The image processing apparatus of claim 1 , wherein the at least one processor is further configured to execute the at least one instruction to:
transform the third feature data divided into third areas comprising a first number of pixels to be divided into the plurality of second patches, and obtain the pieces of fourth feature data by performing the second image processing on each of the plurality of second patches.

7 . The image processing apparatus of claim 1 , wherein the at least one processor is further configured to execute the at least one instruction to:
obtain pieces of first query data, pieces of first key data, and pieces of first value data based on the third feature data, the pieces of first query data, the pieces of first key data, and the pieces of first value data respectively corresponding to third areas comprising a first number of pixels, obtain pieces of second query data, pieces of second key data, and pieces of second value data corresponding to the plurality of second patches, by grouping the pieces of first query data, the pieces of first key data, and the pieces of first value data to respectively correspond to the plurality of second patches, obtain a weight matrix, based on the pieces of second query data and the pieces of second key data, and obtain the pieces of fourth feature data, based on the pieces of second value data and the weight matrix.

8 . The image processing apparatus of claim 1 , wherein the third feature data is obtained from the pieces of second feature data.

9 . The image processing apparatus of claim 1 , wherein the at least one neural network comprises at least one convolutional neural network, and
wherein the at least one processor is further configured to execute the at least one instruction to extract the first feature data from the first image by using the at least one convolutional neural network.

10 . The image processing apparatus of claim 1 , wherein the at least one neural network comprises at least one convolutional neural network, and
wherein the at least one processor is further configured to execute the at least one instruction to:
obtain fifth feature data, based on the pieces of second feature data and the pieces of fourth feature data, and
obtain the second image from the fifth feature data, by using the at least one convolutional neural network.

11 . An operating method of an image processing apparatus for processing an image by using at least one neural network, the operating method comprising:
obtaining first feature data, based on a first image; dividing the first feature data into a plurality of first patches, each of the plurality of first patches comprising M pixels; obtaining pieces of second feature data corresponding to each of pixels included in each of the plurality of first patches, by performing a first self-attention operation on the pixels included in the same patch for each of the plurality of first patches; obtaining third feature data, based on the first image; dividing the pieces of second feature data into a plurality of second patches, each of the plurality of second patches comprising M pixels; obtaining pieces of fourth feature data corresponding to each of the plurality of second patches by performing a second self-attention operation on the plurality of second patches; and generating a second image, based on the pieces of second feature data and the pieces of fourth feature data.

12 . The operating method of claim 11 , wherein the obtaining the pieces of second feature data comprises obtaining the pieces of second feature data respectively corresponding to the plurality of first patches, based on information about peripheral areas of each of the plurality of first patches, and
wherein the obtaining the pieces of fourth feature data comprises obtaining the pieces of fourth feature data respectively corresponding to the plurality of second patches, based on information about peripheral areas of each of the plurality of second patches.

13 . The operating method of claim 11 , wherein each of the plurality of first patches comprises one pixel.

14 . The operating method of claim 11 , wherein the obtaining of the pieces of second feature data comprises:
obtaining pieces of query data, pieces of key data, and pieces of value data based on the first feature data, the pieces of query data, the pieces of key data, and the pieces of value data respectively corresponding to the plurality of first patches; obtaining a weight matrix, based on the pieces of query data and the pieces of key data; and obtaining the pieces of second feature data, based on the pieces of value data and the weight matrix.

15 . The operating method of claim 14 , wherein the obtaining

Assignees

• Samsung Electronics Co Ltd

Inventors

• Kang Soomin
• Song Youngchan
• Lee Tammy
• Park Jaeyeon
• Shin Hanul
• Ahn Iljun

Classifications

• G06T5/70

  Denoising; Smoothing · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20021

  Dividing image into blocks, subimages or windows · CPC title

• G06T2207/20192

  Edge enhancement; Edge preservation · CPC title