Apparatuses and methods for performing artificial intelligence encoding and artificial intelligence decoding on image

Patent metadata

Field	Value
Publication number	US-10817985-B2
Application number	US-201916656812-A
Country	US
Kind code	B2
Filing date	Oct 18, 2019
Priority date	Oct 19, 2018
Publication date	Oct 27, 2020
Grant date	Oct 27, 2020

Abstract

An artificial intelligence (AI) decoding apparatus includes a memory storing one or more instructions, and a processor configured to execute the stored one or more instructions, to obtain image data corresponding to a first image that is encoded, obtain a second image corresponding to the first image by decoding the obtained image data, determine whether to perform AI up-scaling of the obtained second image, based on the AI up-scaling of the obtained second image being determined to be performed, obtain a third image by performing the AI up-scaling of the obtained second image through an up-scaling deep neural network (DNN), and output the obtained third image, and based on the AI up-scaling of the obtained second image being determined to be not performed, output the obtained second image.

First claim

What is claimed is:

1. An electronic device for displaying an image by using an artificial intelligence (AI), the electronic device comprising:
a display; and
one or more processors configured to execute one or more instructions stored in the electronic device, to:
receive AI data that is related to AI down-scaling an original image to a first image through a down-scaling neural network (NN) based on first NN setting information, the first NN setting information being selected from a plurality of first NN setting information that is pre-stored in a server, the AI data comprising a bitrate of an image data generated through an encoding on the first image;
receive the image data;
decode the image data to obtain a second image;
determine whether to perform AI up-scaling through an up-scaling NN, based on the AI data;
based on the AI up-scaling being determined to be performed:
select, based on the AI data, second NN setting information paired with the first NN setting information, from a plurality of second NN setting information that is pre-stored in the electronic device;
obtain a third image corresponding to the original image by performing the AI up-scaling on the obtained second image through the up-scaling NN set with the selected second NN setting information; and
provide, on the display of the electronic device, the obtained third image; and

based on the AI up-scaling of the obtained second image being determined to be not performed:
output the obtained second image,

wherein the plurality of first NN setting information pre-stored in the server and the plurality of second NN setting information pre-stored in the electronic device are obtained through joint training of the up-scaling NN and the down-scaling NN.

2. The electronic device of claim 1 , wherein the one or more processors are further configured to execute the stored one or more instructions to:
identify whether the first image is obtained by performing the AI down-scaling of the original image, based on the AI data; and
based on the first image being identified to be obtained by performing the AI down-scaling of the original image, determine to perform the AI up-scaling of the obtained second image.

3. The electronic device of claim 2 , wherein the one or more processors are further configured to execute the stored one or more instructions to:
identify whether the first image is obtained by performing the AI down-scaling of the original image, based on the AI data, and whether the up-scaling NN is available; and
based on the first image being identified to be obtained by performing the AI down-scaling of the original image, and based on the up-scaling NN being identified to be available, determine to perform the AI up-scaling of the obtained second image.

4. The electronic device of claim 3 , wherein the one or more processors are further configured to execute the stored one or more instructions to:
identify whether the up-scaling NN is stored in the electronic device, and whether second NN setting information for operating the up-scaling NN is stored in the electronic device; and
based on the up-scaling NN being identified to be not stored in the electronic device or the second NN setting information being identified to be not stored in the electronic device, identify that the up-scaling NN is not available.

5. The electronic device of claim 3 , wherein the one or more processors are further configured to execute the stored one or more instructions to, based on the first image being identified to be obtained by performing the AI down-scaling of the original image, and based on the up-scaling NN being identified to be not available, control the electronic device to output the obtained second image, and
wherein the output second image is up-scaled by the display.

6. The electronic device of claim 3 , wherein the one or more processors are further configured to execute the stored one or more instructions to, based on the first image being identified to be obtained by performing the AI down-scaling of the original image, and based on the up-scaling NN being identified to be not available, determine to perform legacy up-scaling of the obtained second image.

7. The electronic device of claim 1 , wherein the AI data comprises at least one of information about a difference between the original image and the first image, a resolution of the first image, a codec type for encoding the first image and the second NN setting information.

8. The electronic device of claim 1 , wherein the up-scaling NN is trained based on quality loss information corresponding to a result of comparing a training image that is output from the up-scaling NN and an original training image.

9. The electronic device of claim 1 , wherein the one or more processors are further configured to execute the stored one or more instructions to obtain the third image by inputting the obtained second image and information related to the obtained second image to the up-scaling NN, and
wherein the information related to the obtained second image comprises either one or both of position information of pixels of the obtained second image and encoding parameter information.

10. The electronic device of claim 1 , wherein the up-scaling NN comprises:
a plurality of convolution layers configured to sequentially perform a plurality of convolution operations on the obtained second image; and
a bypass scaler configured to perform a scaling process on the obtained second image, and
wherein the third image is obtained based on a result of combining an output of the plurality of convolution layers and an output of the bypass scaler.

11. The electronic device of claim 1 , wherein the one or more processors further comprise:
a first processor configured to obtain the second image by decoding the image data; and
a second processor configured to obtain the third image by performing the AI up-scaling,
wherein the second processor is configured to, based on the AI up-scaling of the obtained second image being determined to be performed:
control a first switch so that the second processor obtains, from the first processor, the obtained second image; and
control a second switch so that the electronic device outputs the obtained third image.

12. The electronic device of claim 11 , wherein the second processor is further configured to, based on the AI up-scaling of the obtained second image being determined to be not performed:
control the first switch so that the second processor is prevented from obtaining, from the first processor, the obtained second image; and
control the second switch so that the electronic device outputs the obtained second image.

13. The electronic device of claim 1 , wherein the AI data comprises the information related to the first image,
wherein the one or more processors are further configured to obtain the second NN setting information mapped to the information related to the first image, based on a mapping relationship between a plurality of image-related information and the plurality of second NN setting information, and
wherein the information related to the first image comprises information related to at least one of a resolution and a codec type.

14. The electronic device of claim 1 , wherein the one or more processors are further configured to execute the one or more instructions to:
identify whether the AI data comprises a first flag indicating that the first image is generated through the down-scaling NN or a second flag indicating that the first image is not generated through the down-scaling NN;
determine to perform the AI up-scaling on the obtained sec

Assignees

• Samsung Electronics Co Ltd

Inventors

• Kim Jaehwan
• Lee Jongseok
• Jeon Sunyoung
• CHOI Kwangpyo
• Choi Minseok
• Dinh Quockhanh
• Park Youngo

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/048

  Activation functions · CPC title

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title