Electronic apparatus and controlling method thereof

Patent metadata

Field	Value
Publication number	US-12100118-B2
Application number	US-202117221105-A
Country	US
Kind code	B2
Filing date	Apr 2, 2021
Priority date	Nov 23, 2020
Publication date	Sep 24, 2024
Grant date	Sep 24, 2024

Abstract

An electronic apparatus is disclosed. The electronic apparatus includes a memory configured to store a plurality of neural network models, and a processor connected to the memory and control the electronic apparatus in which the processor is configured to obtain a weight map based on an object area included in an input image, and obtain a plurality of images by inputting the input image to each of the plurality of neural network models, and obtain an output image by weighting the plurality of images based on the weight map, and each of the plurality of neural network models is a model trained to upscale an image.

First claim

What is claimed is:

1. An electronic apparatus comprising:
a memory configured to store a plurality of neural network models; and
a processor connected to the memory and control the electronic apparatus,
wherein the processor is configured to:
obtain a weight map based on an object area included in an input image,
obtain a plurality of images by inputting the input image to each of the plurality of neural network models, and
obtain an output image by weighting the plurality of images based on the weight map,
wherein each of the plurality of neural network models is a model trained to upscale an image, and
wherein the processor is further configured to assign a higher value to an object area included in a first image from among the plurality of images than an object area included in a second image from among the plurality of images, and assign a lower value to a background area included in the first image than a background area included in the second image to weight the first image and the second image.

2. The apparatus of claim 1 , wherein the plurality of neural network models include a first neural network model and a second neural network model,
wherein the processor is configured to:
obtain the first image by inputting the input image to the first neural network model,
obtain the second image by inputting the input image to the second neural network model,
based on the weight map, obtain the output image by weighting the first image and the second image,
wherein the first neural network model is a model in which upscaling processing of the object area is enhanced, and
wherein the second neural network model is a model in which upscaling processing of a background area included in the input image is enhanced.

3. The apparatus of claim 1 , wherein the processor is configured to identify the object area from the input image, and obtain a map in which a weight corresponding to the object area is different from a weight corresponding to the other area included in the input image.

4. The apparatus of claim 3 , wherein the processor is configured to obtain a weight map in which a weight gradually changes based on a center of the object area.

5. The apparatus of claim 1 , wherein the processor is configured to, based on a resolution of the input image being a critical resolution, input the input image to each of first models corresponding to the critical resolution among the plurality of neural network models, and
based on the resolution of the input image being higher than the critical resolution, preprocess the input image, and input the preprocessed image to each of second models among the plurality of neural network models.

6. The apparatus of claim 5 , wherein the processor is configured to, based on the resolution of the input image being higher than the critical resolution, divide the input image into a plurality of sub-images by shuffling, and
input the plurality of sub-images into each of the second models.

7. The apparatus of claim 1 , wherein the processor is configured to, based on a resolution of the input image being a critical resolution, input the input image to each of first models corresponding to the critical resolution among the plurality of neural network models, and
based on the resolution of the input image being higher than the critical resolution, change the input image to an image having the critical resolution by sampling the input image, and input the changed image having the critical resolution to each of the first models.

8. The apparatus of claim 1 , wherein the object area is configured to include at least one of a human body area, a face area, a text area, a graphic area, an artifact area, and a natural object area.

9. The apparatus of claim 1 , wherein the plurality of neural network models include a first neural network model and a second neural network model,
wherein the processor is configured to:
obtain a first image and a second image by inputting the input image to the first neural network model,
obtain a third image by inputting the input image to the second neural network model,
based on the weight map, obtain the output image by weighting the first image, the second image, and the third image,
wherein the first neural network model is a model in which upscaling processing of a first type of an object area included in the input image and corresponding to the first image and upscaling processing of a second type of an object area included in the input image area corresponding to the second image are enhanced,
wherein the second neural network model is a model in which upscaling processing of a third type of an object area included in the input image and corresponding to the third image is enhanced, and
wherein the first type, the second type and the third type are different from each other.

10. A method of controlling an electronic apparatus, the method comprising:
obtaining a weight map based on an object area included in an input image;
obtaining a plurality of images by inputting the input image to each of a plurality of neural network models; and
obtaining an output image by weighting the plurality of images based on the weight map,
wherein each of the plurality of neural network models is a model trained to upscale an image, and
wherein the obtaining the output image includes assigning a higher value to an object area included in a first image from among the plurality of images than an object area included in a second image from among the plurality of images, and assigning a lower value to a background area included in the first image than a background area included in the second image to weight the first image and the second image.

11. The method of claim 10 , wherein the plurality of neural network models include a first neural network model and a second neural network model,
wherein the obtaining the plurality of images includes obtaining the first image by inputting the input image to the first neural network model, and obtaining the second image by inputting the input image to the second neural network model,
wherein the obtaining the output image includes, based on the weight map, obtaining the output image by weighting the first image and the second image,
wherein the first neural network model is a model in which upscaling processing of the object area is enhanced, and
wherein the second neural network model is a model in which upscaling processing of a background area included in the input image is enhanced.

12. The method of claim 10 , wherein the obtaining the weight map includes identifying the object area from the input image, and obtaining a map in which a weight corresponding to the object area is different from a weight corresponding to the other area included in the input image.

13. The method of claim 12 , wherein the obtaining the weight map includes obtaining a weight map in which a weight gradually changes based on a center of the object area.

14. The method of claim 10 , wherein the obtaining the plurality of images includes, based on a resolution of the input image being a critical resolution, inputting the input image to each of first models corresponding to the critical resolution among the plurality of neural network models, and
based on the resolution of the input image being higher than the critical resolution, preprocessing the input image, and inputting the preprocessed image to each of second models among the plurality of neural network models.

15. The method of claim 14 , wherein the inputting the preprocessed image to each of the second models includes, based on the resolution of the input image be

Assignees

• Samsung Electronics Co Ltd

Inventors

• Lee Hyunseung
• Kim Donghyun
• Moon Youngsu
• Park Seungho
• Jeong Younghoon

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title

• G06T2207/20084

  Artificial neural networks [ANN] · CPC title

• G06T2207/20081

  Training; Learning · CPC title