Apparatus and method for performing artificial intelligence encoding and artificial intelligence decoding on image

Patent metadata

Field	Value
Publication number	US-11223841-B2
Application number	US-202117333541-A
Country	US
Kind code	B2
Filing date	May 28, 2021
Priority date	May 29, 2020
Publication date	Jan 11, 2022
Grant date	Jan 11, 2022

Abstract

An artificial intelligence (AI) decoding apparatus includes at least one processor configured to execute one or more instructions to: obtain a second image corresponding to a first image by performing first decoding on image data included in a main bitstream, obtain AI upscaling activation flag information included in AI data of a sub-bitstream, determine whether to perform AI upscaling on the second image, based on the AI upscaling activation flag information, when it is determined that AI upscaling is to be performed on the second image, obtain a third image by performing AI upscaling on the second image, through an upscaling deep neural network (DNN) set according to upscaling DNN information, the upscaling DNN information selected from among a plurality of pieces of pre-stored upscaling DNN information based on at least a portion of the image data and/or at least a portion of AI sub-data.

First claim

What is claimed is:

1. An artificial intelligence (AI) decoding apparatus comprising:
at least one processor configured to execute one or more instructions to:
obtain a bitstream comprising a main bitstream including image data obtained by performing first encoding on a first image, and a sub-bitstream including AI data;
obtain a second image corresponding to the first image by performing first decoding on the image data;
determine whether to perform AI upscaling on the second image, based on AI upscaling activation flag information included in the AI data;
based on determining that the AI upscaling is to be performed on the second image:
obtain a third image by performing the AI upscaling on the second image, through an upscaling deep neural network (DNN) set according to upscaling DNN information, the upscaling DNN information selected from among a plurality of pieces of pre-stored upscaling DNN information based on at least a portion of at least one of the image data or AI sub-data included in the sub-bitstream, and
output the obtained third image; and

based on determining that the AI upscaling is not to be performed on the second image, output the second image,

wherein the AI upscaling activation flag information represents whether to perform the AI upscaling on the second image.

2. The AI decoding apparatus of claim 1 , wherein the AI sub-data comprises at least one of:
image genre information representing the genre of an image;
average quantization parameter information representing an average quantization value of the first image;
quantization parameter indicator information representing a quantization degree of at least a portion of the first image;
image resolution information representing resolution of at least one of an original image or the first image;
image ratio information representing a ratio between the original image and the first image;
codec information representing a codec used during encoding of the first image;
metadata flag information representing whether AI-upscaling related additional metadata is included; or
AI-upscaling related additional metadata information.

3. The AI decoding apparatus of claim 2 , wherein the at least one processor is configured to execute the one or more instructions to set the upscaling DNN with upscaling DNN information from among a plurality of pieces of upscaling DNN information, based on at least one of the image genre information, the average quantization parameter information, the quantization parameter indicator information, the image resolution information, the image ratio information, or the AI-upscaling related additional metadata information.

4. The AI decoding apparatus of claim 3 , wherein:
the quantization parameter indicator information is index information representing one of a plurality of pre-determined quantization parameter groups; and
the at least one processor is configured to execute the one or more instructions to set the upscaling DNN with upscaling DNN information corresponding to a quantization parameter group indicated by the index information from among the plurality of pieces of upscaling DNN information.

5. The AI decoding apparatus of claim 2 , wherein the quantization parameter indicator information is index information representing a quantization parameter group including an average value of a quantization parameter of at least one of a sub-group within a frame, the frame, a frame group, or a video sequence.

6. The AI decoding apparatus of claim 1 , wherein the at least one processor is configured to execute the one or more instructions to set the upscaling DNN with upscaling DNN information selected from among a plurality of pieces of upscaling DNN information, based on quantization parameter information obtained from the image data.

7. The AI decoding apparatus of claim 1 , wherein the main bitstream is included in a predetermined main bitstream region within the bitstream defined in accordance with a predetermined codec, and the sub-bitstream is included in a predetermined sub-bitstream region within the bitstream defined in accordance with the predetermined codec.

8. The AI decoding apparatus of claim 7 , wherein the sub-bitstream is a supplemental enhancement information (SEI) message.

9. The AI decoding apparatus of claim 8 , wherein the SEI message is an SEI message including user data registered by a predetermined standard or including unregistered user data that is identified by a Universally Unique Identifier (UUID).

10. The AI decoding apparatus of claim 2 , wherein the sub-bitstream is a metadata open bitstream unit (OBU).

11. A method of artificial intelligence (AI)-decoding an image, the method comprising:
obtaining a bitstream comprising a main bitstream including image data obtained by performing first encoding on a first image, and a sub-bitstream including AI data;
obtaining a second image corresponding to the first image by performing first decoding on the image data;
obtaining AI upscaling activation flag information included in the AI data;
determining whether to perform AI upscaling on the second image, based on the AI upscaling activation flag information;
based on determining that the AI upscaling is to be performed on the second image:
obtaining a third image by performing the AI upscaling on the second image, through an upscaling deep neural network (DNN) set according to upscaling DNN information, the upscaling DNN information selected from among a plurality of pieces of pre-stored upscaling DNN information based on at least a portion of at least one of the image data or AI sub-data included in the sub-bitstream, and
outputting the obtained third image; and

based on determining that the AI upscaling is not to be performed on the second image, outputting the second image,
wherein the AI upscaling activation flag information represents whether to perform AI upscaling on the second image.

12. The method of claim 11 , wherein the AI sub-data comprises at least one of:
image genre information representing the genre of an image;
average quantization parameter information representing an average quantization value of the first image;
quantization parameter indicator information representing a quantization degree of at least a portion of the first image;
image resolution information representing resolution of at least one of an original image or the first image;
image ratio information representing a ratio between the original image and the first image;
codec information representing a codec used during encoding of the first image;
metadata flag information representing whether AI-upscaling related additional metadata is included; or
AI-upscaling related additional metadata information.

13. The method of claim 12 , wherein the obtaining the third image comprises setting the upscaling DNN with upscaling DNN information from among a plurality of pieces of upscaling DNN information, based on at least one of the image genre information, the average quantization parameter information, the quantization parameter indicator information, the image resolution information, the image ratio information, or the AI-upscaling related additional metadata information.

14. The method of claim 13 , wherein:
the quantization parameter indicator information is index information representing one of a plurality of pre-determined quantization parameter groups; and
setting the upscaling DNN comprises setting the upscaling DNN with upscaling DNN information corresponding to a quantization parameter group indicated by the index information from among the plurality of pieces of upscaling DNN information.

Assignees

• Samsung Electronics Co Ltd

Inventors

• Kim Jaehwan
• Park Youngo
• Lee Chaeeun
• CHOI Kwangpyo

Classifications

• G06N3/048

  Activation functions · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• H04N19/59

  involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution · CPC title