Methods and apparatuses for performing artificial intelligence encoding and artificial intelligence decoding on image

Patent metadata

Field	Value
Publication number	US-10825205-B2
Application number	US-202016793605-A
Country	US
Kind code	B2
Filing date	Feb 18, 2020
Priority date	Oct 19, 2018
Publication date	Nov 3, 2020
Grant date	Nov 3, 2020

Abstract

Provided is an artificial intelligence (AI) decoding apparatus includes: a memory storing one or more instructions; and a processor configured to execute the one or more instructions stored in the memory, the processor is configured to: obtain AI data related to AI down-scaling an original image to a first image; obtain image data corresponding to an encoding result on the first image; obtain a second image corresponding to the first image by performing a decoding on the image data; obtain deep neural network (DNN) setting information among a plurality of DNN setting information from the AI data; and obtain, by an up-scaling DNN, a third image by performing the AI up-scaling on the second image, the up-scaling DNN being configured with the obtained DNN setting information, wherein the plurality of DNN setting information comprises a parameter used in the up-scaling DNN, the parameter being obtained through joint training of the up-scaling DNN and a down-scaling DNN, and wherein the down-scaling DNN is used to obtain the first image from the original image.

First claim

What is claimed is:

1. An electronic device for displaying an image by using an artificial intelligence (AI), the electronic device comprising:
a display; and
one or more processors configured to execute one or more instructions stored in the electronic device, to:
obtain mage data generated through an encoding on a first image;
obtain AI data related to AI down-scaling an original image to the first image, the AI data comprising a bitrate of the image data, and the first image being obtained through the AI down-scaling of the original image by a down-scaling NN configured with first NN setting information selected based on information input by a user from among a plurality of first NN setting information for the AI down-scaling;
obtain a second image by decoding the obtained image data;
select second NN setting information from a plurality of second NN setting information that is pre-stored in the electronic device based on the obtained AI data, the plurality of second NN setting information being for AI up-scaling; and
obtain, by an up-scaling NN, a third image by performing the AI up-scaling on the obtained second image, the up-scaling NN being configured with the selected second NN setting information; and
provide, on the display, the third image,

wherein the plurality of first NN setting information and the plurality of second NN setting information are obtained through joint training of the down-scaling NN and the up-scaling NN.

2. The electronic device of claim 1 , wherein the down-scaling NN is trained based on a spatial complexity of a first training image obtained through the down-scaling NN from an original training image, and the spatial complexity of the first training image comprises a total variance of the first training image.

3. The electronic device of claim 2 , wherein the down-scaling NN is trained based on a comparing result between the first training image and a reduced training image which is legacy down-scaled from the original training image.

4. The electronic device of claim 1 ,
wherein the one or more processors are further configured to select the second NN setting information that is mapped to information about the first image, from a mapping relationship between a plurality of image-related information and the plurality of second NN setting information, the mapping relationship being pre-stored in the electronic device, and
wherein the information about the first image comprises the bitrate of the image data and a codec type.

5. The electronic device of claim 4 , wherein the image data comprises quantization parameter information that is used in the decoding, and
wherein the one or more processors are further configured to select, from the pre-stored mapping relationship, the second NN setting information that is mapped to the quantization parameter information and the information about the first image.

6. The electronic device of claim 1 , wherein the selected second NN setting information comprises parameters of a filter kernel,
wherein the filter kernel is associated with at least one convolution layer, and
wherein the up-scaling NN comprises the at least one convolution layer.

7. The electronic device of claim 1 , wherein the one or more processors are further configured to set the up-scaling NN with the selected second NN setting information instead of second NN setting information that is set in the up-scaling NN, when the second NN setting information set in the up-scaling NN is different from the selected second NN setting information.

8. The electronic device of claim 1 , wherein the up-scaling NN is trained based on quality loss information, and
wherein the quality loss information corresponds to a comparison of a training image that is output from the up-scaling NN and the original training image before the AI down-scaling is performed.

9. The electronic device of claim 8 , wherein the quality loss information is used in training of the down-scaling NN.

10. The electronic device of claim 1 , wherein, when parameters of a first one among the up-scaling NN and the down-scaling NN are updated during the joint training, parameters of a different one among the up-scaling NN and the down-scaling NN are updated.

11. A server for providing an image by using an artificial intelligence (AI), the server comprising:
one or more processors configured to execute one or more instructions stored in the server, to:
select first neural network (NN) setting information from a plurality of first NN setting information that is pre-stored in the server based on information input by a user, the plurality of first NN setting information being for AI down-scaling;
obtain, by a down-scaling NN configured with the selected first NN setting information, a first image by performing the AI down-scaling on an original image;
encode the obtained first image to obtain image data; and
transmit, to an electronic device, the obtained image data and AI data, the AI data comprising a bitrate the image data and being used for selecting second NN setting information of an up-scaling NN from a plurality of second NN setting information that is pre-stored in the electronic device, and the plurality of second NN setting information being for AI up-scaling on a decoded image corresponding to the image data,

wherein the plurality of first NN setting information and the plurality of second NN setting information are obtained through joint training of the down-scaling NN and the up-scaling NN.

12. The server of claim 11 , wherein the down-scaling NN and the up-scaling NN are trained based on quality loss information, and
wherein the quality loss information corresponds to a comparison of a training image that is output from the up-scaling NN and the original training image before the AI down-scaling is performed.

13. A method performed by an electronic device for displaying an image by using an artificial intelligence (AI), the method comprising:
obtaining image data generated through an encoding on a first image;
obtaining AI data related to AI down-scaling an original image to the first image, the AI data comprising a bitrate of the image data, and the first image being obtained through the AI down-scaling of the original image by a down-scaling NN configured with first NN setting information selected based on information input by a user from among a plurality of first NN setting information for the AI down-scaling;
obtaining a second image by decoding the obtained image data;
selecting second NN setting information from a plurality of second NN setting information that is pre-stored in the electronic device based on the obtained AI data, the plurality of second NN setting information being for AI up-scaling; and
obtaining, by an up-scaling NN, a third image by performing the AI up-scaling on the obtained second image, the up-scaling NN being configured with the selected second NN setting information; and
providing, on the display, the third image,
wherein the plurality of first NN setting information and the plurality of second NN setting information are obtained through joint training of the down-scaling NN and the up-scaling NN.

14. A method performed by a server for providing an image by using an artificial intelligence (AI), the method comprising:
selecting first neural network (NN) setting information from a plurality of first NN setting information that is pre-stored in the server based on information input by a user, the plurality of first NN setting information being for AI down-scaling;
obtaining, by a down-scaling NN configured with the selected first NN setting information, a first image by p

Assignees

• Samsung Electronics Co Ltd

Inventors

• Kim Jaehwan
• Lee Jongseok
• Jeon Sunyoung
• CHOI Kwangpyo
• Choi Minseok
• Dinh Quockhanh
• Park Youngo

Classifications

• H04N19/172

  the region being a picture, frame or field · CPC title

• H04N19/117

  Filters, e.g. for pre-processing or post-processing (sub-band filter banks H04N19/635) · CPC title

• H04N19/132 Primary

  Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking · CPC title

• H04N19/85

  using pre-processing or post-processing specially adapted for video compression · CPC title

• G06V10/82

  using neural networks · CPC title