Electronic apparatus and control method thereof

Patent metadata

Field	Value
Publication number	US-12400105-B2
Application number	US-202318216295-A
Country	US
Kind code	B2
Filing date	Jun 29, 2023
Priority date	Jan 26, 2021
Publication date	Aug 26, 2025
Grant date	Aug 26, 2025

Abstract

An electronic apparatus includes a camera including a first lens and a second lens capable of obtaining an image having an angle of view different from the first lens, a display, a memory, and a processor. To perform a controlling method, the processor is configured to provide a first image obtained using the first lens to the display as a live view, obtain a second image using the second lens, obtain image information regarding the second image and object information regarding at least one object depicted in the second image using at least one neural network model, identify a screen type of the second image based on the image information and the object information, identify a set of filters corresponding to the screen type of the second image, and correct the first image based on the identified set of filters to provide a corrected first image as the live view.

First claim

What is claimed is:

1. An electronic apparatus comprising:
a camera including a first lens and a second lens capable of obtaining an image having an angle of view different from the first lens;
a display;
a memory; and
a processor configured to:
provide a first image obtained using the first lens to the display as a live view;
obtain a second image using the second lens while providing the first image as the live view;
obtain image information regarding the second image and object information regarding at least one object depicted in the second image using at least one neural network model;
identify a screen type of the second image based on the image information and the object information;
identify a set of filters corresponding to the screen type of the second image; and
correct the first image based on the identified set of filters to provide a corrected first image as the live view.

2. The apparatus as claimed in claim 1 , wherein the image information comprises depth map information corresponding to the second image and saliency information corresponding to the second image; and
wherein the processor is further configured to obtain the depth map information by inputting the second image to a first neural network model, and obtain the saliency information by inputting the second image to a second neural network model.

3. The apparatus as claimed in claim 2 , wherein the object information comprises at least one of type information of the at least one object, three-dimensional location information of the at least one object, area information regarding an area where the at least one object is located, and posture information of the at least one object; and
wherein the processor is further configured to perform at least one of:
obtain segmentation information in which the at least one object and a background included in the second image are segmented by inputting the second image to a third neural network model;
obtain the type information by inputting the second image to the third neural network model;
obtain at least one of the three-dimensional location information and the area information based on the segmentation information and the depth map information; and
obtain the posture information by inputting information regarding the at least one object included in the segmentation information to a fourth neural network model.

4. The apparatus as claimed in claim 3 , wherein the processor is further configured to obtain relationship information between a plurality of objects depicted in the second image comprising the at least one object, by inputting the three-dimensional location information and the area information of each object of the plurality of objects to a fifth neural network model.

5. The apparatus as claimed in claim 4 , wherein the processor is further configured to obtain heat map information corresponding to corrected saliency information by inputting the relationship information, the posture information of the at least one object, focus information regarding the second image, and the saliency information to a sixth neural network model.

6. The apparatus as claimed in claim 5 , wherein the processor is further configured to identify the screen type of the second image based on the second image, the heat map information, and the type information of the at least one object.

7. The apparatus as claimed in claim 1 , wherein the memory stores a plurality of sets of filters respectively corresponding to a plurality of screen types; and
wherein the processor is further configured to identify the set of filters corresponding to the screen type of the second image from among the plurality of sets of filters.

8. The apparatus as claimed in claim 1 , wherein the processor is further configured to control the display to provide information regarding the screen type on the live view together with the corrected first image.

9. The apparatus as claimed in claim 1 , wherein the second lens is capable of obtaining an image having a wider angle of view than the first lens.

10. A controlling method of an electronic apparatus comprising a camera including a first lens and a second lens capable of obtaining an image having a wider angle of view than the first lens, the method comprising:
providing a first image obtained using the first lens as a live view;
obtaining a second image using the second lens while providing the first image as the live view;
obtaining image information regarding the second image and object information regarding at least one object depicted in the second image using at least one neural network model;
identifying a screen type of the second image based on the image information and the object information;
identifying a set of filters corresponding to the screen type of the second image; and
correcting the first image based on the identified set of filters to provide a corrected first image as the live view.

11. The method as claimed in claim 10 , wherein the image information comprises depth map information corresponding to the second image and saliency information corresponding to the second image; and
wherein the obtaining of the image information comprises obtaining the depth map information by inputting the second image to a first neural network model, and obtaining the saliency information by inputting the second image to a second neural network model.

12. The method as claimed in claim 11 , wherein the object information comprises at least one of type information of the at least one object, three-dimensional location information of the at least one object, area information regarding an area where the at least one object is located, and posture information of the at least one object; and
wherein the obtaining of the object information comprises at least one of:
obtaining segmentation information in which the at least one object and a background included in the second image are segmented by inputting the second image to a third neural network model;
obtaining the type information by inputting the second image to the third neural network model;
obtaining at least one of the three-dimensional location information and the area information based on the segmentation information and the depth map information; and
obtaining the posture information by inputting information regarding the at least one object included in the segmentation information to a fourth neural network model.

13. The method as claimed in claim 12 , wherein the identifying of the screen type of the second image comprises obtaining relationship information between a plurality of objects depicted in the second image comprising the at least one object, by inputting the three-dimensional location information and the area information of each object of the plurality of objects to a fifth neural network model.

14. The method as claimed in claim 13 , wherein the identifying of the screen type of the second image comprises obtaining heat map information corresponding to corrected saliency information by inputting the relationship information, the posture information of the at least one object, focus information regarding the second image, and the saliency information to a sixth neural network model.

15. The method as claimed in claim 14 , wherein the identifying of the screen type of the second image is based on the second image, the heat map information, and the type information of the at least one object.

16. The method as claimed in claim 10 , wherein the memory stores a plurality of sets of filters respectively corresponding to a plurality of screen types, and
wherein the identify

Assignees

• Samsung Electronics Co Ltd

Inventors

• Oh Seongwoo
• Yoon Yeonggyo
• Hwang Jinyoung

Classifications

• H04N23/55

  Optical parts specially adapted for electronic image sensors; Mounting thereof · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06T5/00

  Image enhancement or restoration · CPC title

• H04N23/632 Primary

  for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters · CPC title