AI encoding apparatus and operation method of the same, and AI decoding apparatus and operation method of the same

Patent metadata

Field	Value
Publication number	US-11200702-B2
Application number	US-202117147954-A
Country	US
Kind code	B2
Filing date	Jan 13, 2021
Priority date	Oct 19, 2018
Publication date	Dec 14, 2021
Grant date	Dec 14, 2021

Abstract

Provided is an artificial intelligence (AI) decoding apparatus including a memory storing one or more instructions; and a processor configured to execute the one or more instructions to, when an image is input to a second DNN including a plurality of layers, obtain first result values based on an operation between the image and a first filter kernel and obtain second result values based on an operation between the image and a second filter kernel, from a first layer including the first and second filter kernels from among the plurality of layers, perform normalization by transforming the first result values into first values by using a first scale factor, and, perform normalization by transforming the second result values into second values by using a second scale factor, transform the first values and the second values into integer values included in a preset range.

First claim

What is claimed is:

1. An electronic device for displaying an image by using an artificial intelligence (AI), the electronic device comprising:
a display; and
one or more processors configured to:
obtain AI data related to AI down-scaling an original image to a first image, the AI data comprising an index indicating neural network (NN) setting information for an AI up-scaling,
obtain image data corresponding to an encoding result on the first image,
obtain a second image by decoding the obtained image data,
select NN setting information from a plurality of NN setting information, the NN setting information corresponding to at least a portion of the obtained AI data, and the NN setting information being for performing the AI up-scaling on the obtained second image,
input the second image to an up-scaling NN comprising a plurality of layers comprising a first layer and a second layer, the first layer comprising a first filter kernel and a second filter kernel, the up-scaling NN being set with the selected NN setting information,
obtain first result values based on an operation between the second image and the first filter kernel, and obtain second result values based on an operation between the second image and the second filter kernel,
perform normalization by transforming the first result values into first values by using a first scale factor, and perform normalization by transforming the second result values into second values by using a second scale factor,
transform the first values into first integer values included in a preset range and transform the second values into second integer values included in the preset range,
input the first and second integer values to the second layer,
obtain third result values based on the first and second integer values in the second layer, and obtain a third image by up-scaling the second image, based on the third result values, and
provide, on the display of the electronic device, the obtained third image,

wherein the first scale factor and the second scale factor are set in association with scale factors of a down-scaling NN.

2. The electronic device of claim 1 , wherein the up-scaling NN is a network trained in connection with the down-scaling NN, based on an image obtained during training of the down-scaling NN, and
the first scale factor and the second scale factor are set via training of the up-scaling NN and the down-scaling NN.

3. The electronic device of claim 1 , wherein the first scale factor and the second scale factor are trained such that a normal distribution of the first values becomes equal to a normal distribution of the second values.

4. The electronic device of claim 1 , wherein the one or more processors are further configured to clip the first values and the second values in the preset range and transform the clipped first values into the first integer values and transform the clipped second values into the second integer values.

5. The electronic device of claim 4 , wherein the one or more processors are further configured to transform the clipped first values into the first integer values and transform the clipped second values into the second integer values, by rounding off the clipped first and second values.

6. The electronic device of claim 1 , wherein the one or more processors are further configured to store the first and second integer values in a memory.

7. The electronic device of claim 6 , wherein a size of the memory that stores the first and second integer values is determined based on the preset range.

8. The electronic device of claim 1 , wherein the one or more processors are further configured to:
generate a first feature map by performing a convolution operation between the second image and the first filter kernel, and obtain the first result values by applying a first activation function to the first feature map, and
generate a second feature map by performing a convolution operation between the second image and the second filter kernel, and obtain the second result values by applying a second activation function to the second feature map.

9. The electronic device of claim 8 , wherein
the first activation function and the second activation function are expressed as

f
⁡

(
x
)

=

{

ax
,

x
<
0

bx
,

x
≥
0

,

wherein f(x) denotes the first activation function or the second activation function, a and b denote slopes that have different values, and
wherein a value of b is determined via training of the up-scaling NN.

10. The electronic device of claim 9 , wherein the value of b in the first activation function is different from the value of b in the second activation function.

11. The electronic device of claim 1 , wherein the AI data comprises information related to at least one of a bitrate regarding the image data, a quantization parameter regarding the image data, a resolution of the first image, or a codec type used in the encoding the first image.

12. A server for providing an image by using an artificial intelligence (AI) comprising:
a memory storing one or more instructions; and
one or more processors configured to execute the stored one or more instructions to:
select neural network (NN) setting information for an AI down-scaling from a plurality of NN setting information,
input an original image to a down-scaling NN comprising a plurality of layers comprising a first layer and a second layer, the first layer comprising a first filter kernel and a second filter kernel, the down-scaling NN being set with the selected NN setting information,
obtain first result values based on an operation between the original image and the first filter kernel, and obtain second result values based on an operation between the original image and the second filter kernel,
perform normalization by transforming the first result values into first values by using a first scale factor, and perform normalization by transforming the second result values into second values by using a second scale factor,
transform the first values into first integer values included in a preset range and transform the second values into second integer values included in the preset range,
input the first and second integer values to the second layer, and obtain third result values based on the first and second integer values in the second layer,
obtain a first image by downscaling the original image, based on the third result values,
encode the first image to obtain image data, and
provide an electronic device with the image data and AI d

Assignees

• Samsung Electronics Co Ltd

Inventors

• Dinh Quockhanh
• Choi Minseok
• CHOI Kwangpyo

Classifications

• G06V10/82

  using neural networks · CPC title

• G06V10/7715

  Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title

• G06T9/002 Primary

  using neural networks · CPC title

• G06T3/4046 Primary

  using neural networks · CPC title