Method and electronic device for capturing media using under display camera

Patent metadata

Field	Value
Publication number	US-12118693-B2
Application number	US-202217950331-A
Country	US
Kind code	B2
Filing date	Sep 22, 2022
Priority date	Jul 21, 2021
Publication date	Oct 15, 2024
Grant date	Oct 15, 2024

Abstract

An electronic device includes a UDC and a UDC controller configured to determine an optimal number of frames required to be captured for a scene to compensate at least one parameter to optimize an output media using the UDC, obtain a multi-frame fusion media by performing at least one multi-frame fusion on the determined optimal number of frames, perform a light source spread correction on the multi-frame fusion media, and optimize the output media based on the light source spread correction on the multi-frame fusion media.

First claim

What is claimed is:

1. An electronic device comprising:
an under display camera (UDC); and
a UDC controller configured to:
determine an optimal number of frames required to be captured for a scene to compensate at least one parameter to optimize an output media using the UDC;
obtain a multi-frame fusion media by performing at least one multi-frame fusion on the determined optimal number of frames;
perform a light source spread correction on the multi-frame fusion media; and
optimize the output media based on the light source spread correction on the multi-frame fusion media.

2. The electronic device of claim 1 , wherein the UDC controller is further configured to determine the optimal number of frames by:
receiving at least one input frame and a metadata information of the scene to be captured using the UDC;
measuring at least one ambient light parameter of the scene;
determining a light transmission loss through a display panel associated with the UDC by analyzing at least one input frame in connection with at least one ambient light parameter;
identifying at least one intrinsic camera parameter associated with the at least one input frame and a type of the scene; and
determining the optimal number of frames required to be captured for the scene to compensate for the light transmission loss and the at least one ambient light parameter based on the at least one intrinsic camera parameter associated with the at least one input frame and the type of the scene.

3. The electronic device of claim 2 , wherein the at least one intrinsic camera parameter comprises at least one of an exposure time, a gain, a color, temperature and a focus region, and
wherein the type of the scene comprises at least one of an indoor environment, an outdoor environment, a mix lighting condition, a bright day light, and low light condition.

4. The electronic device of claim 2 , wherein the UDC controller is further configured to:
determine at least one display panel characteristic associated with the UDC for capturing each frame among the optimal number of frames;
apply the measured light transmission loss, the at least one ambient light parameter; and the display panel characteristics for capturing each frame among the optimal number of frames; and
automatically modify at least one media capture setting of the UDC by applying the measured light transmission loss, the at least one ambient light parameter and the at least one display panel characteristic.

5. The electronic device of claim 4 , wherein the UDC controller is further configured to determine the at least one display panel characteristic associated with the UDC by:
obtaining at least one intrinsic camera parameter associated with the at least one input frame; and
determining the at least one display panel characteristic associated with the UDC based on the at least one intrinsic camera parameter.

6. The electronic device of claim 4 , wherein the at least one display panel characteristic comprises a diffraction pattern, a point spread function of the display panel, and a modulation transfer function (MTF) of the display panel.

7. The electronic device of claim 2 , wherein the UDC controller is further configured to determine the light transmission loss by:
analyzing at least one preview frame from the UDC, and
estimating the light transmission loss for the display panel based on the analyzing of the at least one preview frame.

8. The electronic device of claim 1 , wherein the UDC controller is further configured to obtain the multi-frame fusion media by:
determining whether a dominant light source is present in the scene;
performing at least one of:
analyzing the scene to determine the multi-frame fusion media to be formed with at least one input frame with multiple exposure values upon determining the dominant light source is present in the scene, or
analyzing the scene to determine the multi-frame fusion media to be formed with the at least one input frame with a same exposure value upon determining the dominant light source is not present in the scene;

estimating a diffraction model on the multi-frame fusion media using an artificial intelligence (AI) model; and
performing an inversion operation to compensate for a diffraction loss in the multi-frame fusion media using the estimated diffraction model.

9. The electronic device of claim 8 , wherein the diffraction model is estimated on the multi-frame fusion media based on transmission characteristics of a display panel and a light transmission loss.

10. The electronic device of claim 1 , wherein the UDC controller is further configured to perform the light source spread correction on the multi-frame fusion media by:
analyzing the multi-frame fusion media to determine a type of the scene and localize a light source in the scene;
creating a segmentation mask for the localized light source in the scene;
applying a light spread correction for the localized light source in the scene using the segmentation mask; and
performing a content aware sharpness improvement to enhance the output media based on the applied light spread correction.

11. The electronic device of claim 10 , wherein the light spread correction is applied for the localized light source in the scene using an artificial intelligence (AI) model, and wherein the AI model is configured to learn a light spread function of a display panel over a period of time.

12. The electronic device of claim 1 , wherein the at least one parameter comprises at least one of a light transmission loss associated with the scene, an ambient light parameter associated with the scene, a display panel characteristic associated with the UDC, an exposure time associated with at least one input frame, a gain associated with the at least one input frame, a temperature associated with the at least one input frame and focus region, and an environment lighting condition.

13. An electronic device comprising:
an under display camera (UDC); and
a UDC controller configured to:
receive at least one input frame and a metadata information of a scene to be captured using the UDC;
measure at least one ambient light parameter of the scene;
determine a light transmission loss through a display panel associated with the UDC by analyzing the at least one input frame in connection with at least one measured ambient light parameter;
identify at least one intrinsic camera parameter associated with the at least one input frame and a type of the scene;
determine an optimal number of frames required to be captured for the scene to compensate for the light transmission loss and at least one ambient light parameter based on the at least one intrinsic camera parameter associated with the at least one input frame and the type of the scene;
determine at least one display panel characteristic associated with the UDC;
apply the measured light transmission loss, the at least one ambient light parameter and the display panel characteristics for capturing each frame among the optimal number of frames;

automatically modify at least one media capture setting of the UDC by applying the measured light transmission loss, the at least one ambient light parameter and the at least one display panel characteristic; and
optimize a media capture for the UDC in the electronic device based on the at least one modified media capture setting of the UDC.

14. A method for capturing media using an under display camera (UDC) in an electronic device, comprising:
determining, by the electronic device, an optimal number of frames required to be captured for a scene to compensate at least one parameter to optimize an ou

Assignees

• Samsung Electronics Co Ltd

Inventors

• Shukla Alok Shankarlal
• Kumar Manoj
• Sharma Saketh
• Choi Woojhon
• Do Wonjoon
• Choi Jaesung

Classifications

• G06T5/80

  Geometric correction · CPC title

• G06T5/73

  Deblurring; Sharpening · CPC title

• H04N23/741

  by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors · CPC title

• H04N23/72

  Combination of two or more compensation controls · CPC title

• H04N23/71

  Circuitry for evaluating the brightness variation · CPC title