Method and apparatus for performing artificial intelligence encoding and artificial intelligence decoding

Patent metadata

Field	Value
Publication number	US-11270469-B2
Application number	US-202117333845-A
Country	US
Kind code	B2
Filing date	May 28, 2021
Priority date	Jun 11, 2020
Publication date	Mar 8, 2022
Grant date	Mar 8, 2022

Abstract

An apparatus for performing artificial intelligence (AI) encoding on an image includes: a memory storing one or more instructions; and a processor configured to execute the one or more instructions stored in the memory to: determine a resolution of an original image; when the resolution of the original image is higher than a predetermined value, obtain a first image by performing AI downscaling on the original image via a downscaling deep neural network (DNN); when the resolution of the original image is lower than or equal to the predetermined value, obtain a first image by performing AI one-to-one preprocessing on the original image via a one-to-one preprocessing DNN for upscaling; generate image data by performing first encoding on the first image; and transmit the image data and AI data including information related to the AI downscaling or information related to the AI one-to-one preprocessing.

First claim

What is claimed is:

1. An apparatus for performing artificial intelligence (AI) encoding on an image, the apparatus comprising:
a memory storing one or more instructions; and
at least one processor configured to execute the one or more instructions stored in the memory to:
determine a resolution of an original image;
based on the resolution of the original image being greater than a predetermined value, obtain a first image by performing AI downscaling on the original image via a downscaling deep neural network (DNN);
based on the resolution of the original image being less than or being equal to the predetermined value, obtain the first image by performing AI one-to-one preprocessing on the original image via a one-to-one preprocessing DNN for upscaling;
generate image data by performing first encoding on the first image; and
transmit the image data and AI data including information related to the AI downscaling or information related to the AI one-to-one preprocessing,

wherein the AI data comprises information for selecting DNN setting information of an upscaling DNN for AI upscaling of a second image that is generated by performing first decoding on the image data,
wherein DNN setting information of the downscaling DNN is obtained via first joint training of the downscaling DNN and the upscaling DNN, and
wherein DNN setting information of the one-to-one preprocessing DNN is obtained via second joint training of the one-to-one preprocessing DNN and the upscaling DNN by using the DNN setting information of the upscaling DNN obtained via the first joint training.

2. The apparatus of claim 1 , wherein the one-to-one preprocessing DNN is configured to enhance features of the original image while maintaining a same resolution as that of the original image.

3. The apparatus of claim 1 , wherein the downscaling DNN and the upscaling DNN are jointly trained based on first loss information corresponding to a result of comparing a first training image output from the upscaling DNN with an original training image that has not undergone the AI downscaling.

4. The apparatus of claim 3 , wherein the downscaling DNN is trained by further taking into account at least one of second loss information, which corresponds to a result of comparing a reduced training image corresponding to the original training image with a second training image output from the downscaling DNN, or third loss information corresponding to a degree of spatial complexity of the second training image.

5. The apparatus of claim 1 , wherein the one-to-one preprocessing DNN is trained by taking into account fourth loss information corresponding to a result of comparing the original training image with a fourth training image obtained via the one-to-one preprocessing DNN and the upscaling DNN by taking, as an input, a reduced training image corresponding to the original training image.

6. The apparatus of claim 1 , wherein the predetermined value is a 2K resolution.

7. The apparatus of claim 1 , wherein the predetermined value is a 4K resolution.

8. The apparatus of claim 1 , wherein, based on a parameter of one of the upscaling DNN and the downscaling DNN being updated during the first joint training, a corresponding parameter of the other one of the upscaling DNN and the downscaling DNN is updated accordingly.

9. The apparatus of claim 8 , wherein, based on a parameter of the upscaling DNN being updated during the first joint training, a corresponding parameter of the one-to-one preprocessing DNN is updated accordingly during the second joint training.

10. A method of performing artificial intelligence (AI) encoding on an image, the method comprising:
determining a resolution of an original image;
determining, based on the resolution of the original image, whether to obtain a first image by performing AI one-to-one processing on the original image via a one-to-one preprocessing deep neural network (DNN) for upscaling, or by performing AI downscaling on the original image via a downscaling DNN;
based on the resolution of the original image being less than or being equal to a predetermined value, obtaining the first image by performing the AI one-to-one preprocessing on the original image via the one-to-one preprocessing DNN for upscaling;
based on the resolution of the original image being greater than the predetermined value, obtaining the first image by performing the AI downscaling on the original image via the downscaling DNN;
generating image data by performing first encoding on the first image; and
transmitting the image data and AI data including information related to the AI one-to-one preprocessing or information related to the AI downscaling,
wherein the AI data comprises information for selecting DNN setting information of an upscaling DNN for AI upscaling of a second image that is generated by performing first decoding on the image data,
wherein DNN setting information of the downscaling DNN is obtained via first joint training of the downscaling DNN and the upscaling DNN, and
wherein DNN setting information of the one-to-one preprocessing DNN is obtained via second joint training of the one-to-one preprocessing DNN and the upscaling DNN by using the DNN setting information of the upscaling DNN obtained via the first joint training.

11. The method of claim 10 , wherein the one-to-one preprocessing DNN is configured to enhance features of the original image while maintaining a same resolution as that of the original image.

12. The method of claim 10 , wherein the downscaling DNN and the upscaling DNN are jointly trained based on first loss information corresponding to a result of comparing a first training image output from the upscaling DNN with an original training image that has not undergone the AI downscaling.

13. The method of claim 12 , wherein the downscaling DNN is trained by further taking into account at least one of second loss information, which corresponds to a result of comparing a reduced training image corresponding to the original training image with a second training image output from the downscaling DNN, or third loss information corresponding to a degree of spatial complexity of the second training image.

14. The method of claim 10 , wherein the one-to-one preprocessing DNN is trained by taking into account fourth loss information corresponding to a result of comparing the original training image with a fourth training image obtained via the one-to-one preprocessing DNN and the upscaling DNN by taking, as an input, a reduced training image corresponding to the original training image.

15. The method of claim 10 , wherein the predetermined value is a 2K resolution or a 4K resolution.

16. A non-transitory computer-readable recording medium having recorded thereon a program for performing the method of claim 10 .

Assignees

• Samsung Electronics Co Ltd

Inventors

• Lee Chaeeun
• Kim Jaehwan
• Park Youngo

Classifications

• G06N3/048

  Activation functions · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06V10/454

  Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN] · CPC title