Apparatus and method of using AI metadata related to image quality

Patent metadata

Field	Value
Publication number	US-11636626-B2
Application number	US-202017100212-A
Country	US
Kind code	B2
Filing date	Nov 20, 2020
Priority date	Nov 20, 2019
Publication date	Apr 25, 2023
Grant date	Apr 25, 2023

Abstract

An image providing apparatus configured to generate, by using a first artificial intelligence (AI) network, AI metadata including class information and at least one class map, in which the class information includes at least one class corresponding to a type of an object among a plurality of predefined objects included in a first image and the at least one class map indicates a region corresponding to each class in the first image, generate an encoded image by encoding the first image, and output the encoded image and the AI metadata through the output interface.

First claim

What is claimed is:

1. An image providing apparatus comprising:
a memory storing one or more instructions;
one or more processors configured to execute the one or more instructions stored in the memory; and
an output interface,
wherein the one or more processors, by executing the one or more instructions, are configured to:
generate, by using a first artificial intelligence (AI) network, AI metadata comprising class information and at least one class map, wherein the class information comprises at least one class, each of the at least one class corresponds to a type of an object included in a first image among a plurality of predefined objects, and each of the at least one class map indicates regions corresponding to each class among the at least one class in the first image;
generate an encoded image by encoding the first image; and
output the encoded image and the AI metadata through the output interface.

2. The image providing apparatus of claim 1 , wherein the one or more processors, by executing the one or more instructions, are configured to:
input the first image to the first AI network and generate a plurality of segmentation probability maps for types of the plurality of predefined objects;
define the at least one class based on the plurality of segmentation probability maps;
generate the class information comprising the at least one class; and
generate the at least one class map for each class of the at least one class from the plurality of segmentation probability maps.

3. The image providing apparatus of claim 2 , wherein the one or more processors, by executing the one or more instructions, are configured to:
calculate an average value of pixels except for at least one pixel having a value of 0 for each of the plurality of segmentation probability maps; and
select a subset of objects from among the plurality of predefined objects and define the at least one class, based on a magnitude of the average value of each of the plurality of segmentation probability maps.

4. The image providing apparatus of claim 3 , wherein the one or more processors, by executing the one or more instructions, are configured to:
map at least some objects from among objects other than the subset of objects among the plurality of predefined objects to the at least one class; and
generate the at least one class map by combining the plurality of segmentation probability maps.

5. The image providing apparatus of claim 1 , wherein the one or more processors, by executing the one or more instructions, are configured to:
generate a segmentation probability map for a frequency for each of a plurality of predefined frequency domains from the first AI network; and
generate AI metadata based on the segmentation probability map for the frequency, in which the AI metadata comprises frequency information, which comprises information about a frequency domain for the first image, and at least one frequency map corresponding to each frequency domain included in the frequency information.

6. The image providing apparatus of claim 1 , wherein the first image comprises a plurality of images, and the class information and the at least one class map are generated for each of the plurality of images, and
the one or more processors, by executing the one or more instructions, are configured to:
define at least one sequence comprising at least one image from the plurality of images; and
generate, for each sequence among the at least one sequence, sequence class information indicating information about a class included in the at least one image included in the sequence, and frame class information indicating information about a class included in each of the at least one image included in the sequence, in which the frame class information indicates a combination of classes included in a frame among classes included in the sequence class information and comprises a number of bits less than the sequence class information.

7. The image providing apparatus of claim 1 , wherein the one or more processors, by executing the one or more instructions, are configured to:
generate, based on the at least one class map, a lightweighted class map in which each pixel has a representative value corresponding to a class among the at least one class;
generate lightweighted AI metadata comprising the class information and the lightweighted class map; and
output the encoded image and the lightweighted class map through the output interface.

8. The image providing apparatus of claim 1 , wherein the first AI network comprises:
a first-sub AI network comprising at least one convolution layer and at least one maximum pooling layer, and configured to generate a feature map from the first image; and
a second-sub AI network comprising a first layer group that comprises at least one convolution layer and at least one activation layer and receives and processes the feature map from the first-sub AI network, an up-scaler configured to upscale an output of the first layer group, and a second layer group that comprises at least one convolution layer and at least one minimum pooling layer and receives an output of the up-scaler and generates a segmentation probability map for each object of the plurality of predefined objects.

9. The image providing apparatus of claim 1 , wherein the first AI network is trained jointly with a second AI network, and
the second AI network is included in a device configured to receive the encoded image and the AI metadata and decode the encoded image, and perform image-quality processing on image data of the AI metadata, the image data corresponding to the encoded image.

10. The image providing apparatus of claim 1 , wherein the one or more processors, by executing the one or more instructions, are configured to generate the encoded image by down-scaling and encoding the first image.

11. A control method for an image providing apparatus, the control method comprising:
generating, by using a first artificial intelligence (AI) network, AI metadata comprising class information and at least one class map, wherein the class information comprises at least one class, each of the at least one class corresponds to a type of an object included in a first image among a plurality of predefined objects, and each of the at least one class map indicates regions corresponding to each class among the at least one class in the first image;
generating an encoded image by encoding the first image; and
outputting the encoded image and the AI metadata.

12. The control method of claim 11 , further comprising:
inputting the first image to the first AI network and generating a plurality of segmentation probability maps for types of the plurality of predefined objects;
defining at least one class based on the plurality of segmentation probability maps;
generating the class information comprising the at least one class; and
generating the at least one class map for each class of the at least one class from the plurality of segmentation probability maps.

13. The control method of claim 12 , further comprising:
calculating an average value of pixels except for at least one pixel having a value of 0 for each of the plurality of segmentation probability maps; and
selecting a subset of objects from among the plurality of predefined objects and defining the at least one class, based on a magnitude of the average value of each of the plurality of segmentation probability maps.

14. The control method of claim 13 , further comprising:
mapping at least some objects from among objects other than the subset of objects among the plurality of predefined objects to the

Assignees

• Samsung Electronics Co Ltd

Inventors

• Baek Sangwook
• Cheon Minsu
• Park Yongsup
• Park Jaeyeon
• CHOI Kwangpyo

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0475

  Generative networks · CPC title

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/094

  Adversarial learning · CPC title