Image AI-coding method and device, and image AI-decoding method and device

Patent metadata

Field	Value
Publication number	US-12475528-B2
Application number	US-202217696518-A
Country	US
Kind code	B2
Filing date	Mar 16, 2022
Priority date	Sep 17, 2019
Publication date	Nov 18, 2025
Grant date	Nov 18, 2025

Abstract

An AI decoding apparatus includes a memory storing instructions and a processor configured to execute the instructions to obtain AI data related to AI down-scaling of an original image and image data generated as a result of encoding a first image, obtain a second image corresponding to the first image by decoding the image data, determine a resolution ratio in a horizontal direction and a resolution ratio in a vertical direction between the original image and the first image, based on the AI data, and obtain, by an up-scaling deep neural network (DNN), a third image in which a resolution in at least one of a horizontal direction and a vertical direction is increased from the second image based on the resolution ratio in the horizontal direction and the resolution ratio in the vertical direction.

First claim

What is claimed is:

1 . An artificial intelligence (AI) decoding apparatus comprising:
a memory storing instructions; and a processor configured to execute the instructions to:
obtain AI data related to AI down-scaling of an original image and image data generated as a result of encoding a first image,
obtain a second image corresponding to the first image by decoding the image data,
determine a resolution ratio in a horizontal direction and a resolution ratio in a vertical direction between the original image and the first image, based on the AI data, and
obtain, by an up-scaling deep neural network (DNN), a third image in which a resolution in at least one of a horizontal direction and a vertical direction is increased from the second image based on the resolution ratio in the horizontal direction and the resolution ratio in the vertical direction,
wherein the resolution ratio in the horizontal direction and the resolution ratio in the vertical direction are determined as different values.

2 . The AI decoding apparatus of claim 1 , wherein the processor is further configured to execute the instructions to:
obtain, from the second image, a plurality of first intermediate images having a resolution lower than a resolution of the second image, obtain a plurality of second intermediate images output from the up-scaling DNN, based on the plurality of first intermediate images, and obtain the third image having a resolution greater than a resolution of the plurality of second intermediate images by combining the plurality of second intermediate images.

3 . The AI decoding apparatus of claim 2 , wherein the processor is further configured to execute the instructions to obtain the plurality of first intermediate images including a number of pixel lines from among a plurality of pixel lines included in the second image.

4 . The AI decoding apparatus of claim 2 , wherein the processor is further configured to execute the instructions to obtain the third image by alternately connecting pixels included in the plurality of second intermediate images.

5 . The AI decoding apparatus of claim 2 , wherein the processor is further configured to execute the instructions to:
determine a number of the plurality of first intermediate images and a number of the plurality of second intermediate images, based on the resolution ratio in the horizontal direction and the resolution ratio in the vertical direction, and obtain DNN setting information that allow the up-scaling DNN to output the determined number of the plurality of second intermediate images by processing the determined number of the plurality of first intermediate images.

6 . The AI decoding apparatus of claim 5 , wherein the up-scaling DNN comprises a plurality of convolution layers, and
based on the obtained DNN setting information being set in the up-scaling DNN, a number of filter kernels of a last convolution layer from among the plurality of convolution layers is determined to be the same as the number of the plurality of second intermediate images.

7 . The AI decoding apparatus of claim 2 , wherein the processor is further configured to execute the instructions to:
determine a number of operations of the up-scaling DNN based on the resolution ratio in the horizontal direction and the resolution ratio in the vertical direction, and obtain the third image by combining the plurality of second intermediate images output from the up-scaling DNN as a result of operations performed based on the number of operations, wherein the up-scaling DNN outputs a pre-determined number of the plurality of second intermediate images by processing a pre-determined number of the plurality of first intermediate images.

8 . The AI decoding apparatus of claim 7 , wherein,
based on the up-scaling DNN operating to increase one of a resolution of the second image in a horizontal direction and a resolution of the second image in a vertical direction by n times, the processor is further configured to execute the instructions to determine the number of operations of the up-scaling DNN to be a+b when the resolution ratio in the horizontal direction is 1/n a and the resolution ratio in the vertical direction is 1/n b , wherein n is a natural number, and wherein a and b are each an integer equal to or greater than 0.

9 . The AI decoding apparatus of claim 7 , wherein the processor is further configured to execute the instructions to:
combine the plurality of second intermediate images obtained as a result of a previous operation of the up-scaling DNN while the up-scaling DNN operates based on the determined number of operations, and input the plurality of first intermediate images obtained from an image to which the plurality of second intermediate images are combined to the up-scaling DNN.

10 . The AI decoding apparatus of claim 1 , wherein the processor is further configured to execute the instructions to:
scale the second image, and obtain a final third image by adding the scaled second image and the third image.

11 . An artificial intelligence (AI) decoding apparatus comprising:
a memory storing instructions; and a processor configured to execute the instructions to:
obtain AI data related to AI down-scaling of an original image and image data generated as a result of encoding a first image,
obtain a second image corresponding to the first image by decoding the image data,
determine a resolution ratio in a horizontal direction and a resolution ratio in a vertical direction between the original image and the first image, based on the AI data, and
obtain, by an up-scaling deep neural network (DNN), a third image in which a resolution in at least one of a horizontal direction and a vertical direction is increased from the second image based on the resolution ratio in the horizontal direction and the resolution ratio in the vertical direction,
wherein the resolution ratio in the horizontal direction and the resolution ratio in the vertical direction are determined as different values, and wherein the processor is further configured to obtain, based on information included in the AI data, DNN setting information corresponding to the up-scaling DNN for up-scaling the second image to the third image.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Kim Jaehwan
• Lee Jongseok
• Park Youngo
• Lee Chaeeun

Classifications

• G06T3/4053

  based on super-resolution, i.e. the output image resolution being higher than the sensor resolution · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/08

  Learning methods · CPC title

• H04N19/59

  involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution · CPC title

• H04N19/423

  characterised by memory arrangements (H04N19/433 takes precedence) · CPC title