Method and device for balancing foreground-background luminosity

Patent metadata

Field	Value
Publication number	US-10834329-B2
Application number	US-201815978122-A
Country	US
Kind code	B2
Filing date	May 12, 2018
Priority date	Jun 2, 2017
Publication date	Nov 10, 2020
Grant date	Nov 10, 2020

Abstract

In one embodiment, a method includes: obtaining a first image of a scene while an illumination component is set to an inactive state; obtaining a second image of the scene while the illumination component is set to a pre-flash state; determining one or more illumination control parameters for the illumination component for a third image of the scene that satisfy a foreground-background balance criterion based on a function of the first and second images in order to discriminate foreground data from background data within the scene; and obtaining the third image of the scene while the illumination component is set to an active state in accordance with the one or more illumination control parameters.

First claim

What is claimed is:

1. A method comprising:
at a device with one or more processors, non-transitory memory, an image sensor, and an illumination component:
obtaining, by the image sensor, a first image of a scene while the illumination component is set to an inactive state;
obtaining, by the image sensor, a second image of the scene while the illumination component is set to a pre-flash state;
generating a foreground estimation of the scene based on relative brightness differences of pixels of the first image and pixels of the second image;
generating a delta luminosity histogram based on the foreground estimation;
determining one or more illumination control parameters for the illumination component for a third image of the scene that satisfy a foreground-background balance criterion based on the delta luminosity histogram; and
obtaining, by the image sensor, the third image of the scene while the illumination component is set to an active state in accordance with the one or more illumination control parameters.

2. The method of claim 1 , wherein the one or more illumination control parameters set a flash period of the illumination component for the third image relative to a predefined exposure period of the third image.

3. The method of claim 1 , further comprising, determining one or more image parameters for the image sensor for the third image based on the foreground estimation in order to discriminate the foreground data from the background data within the scene, wherein one of the one or more image parameters sets an exposure period for the third image, and
wherein the one or more illumination control parameters set a flash period of the illumination component for the third image relative to the exposure period of the third image.

4. The method of claim 1 , wherein the one or more illumination control parameters correspond to one of a flash timing parameter relative to an exposure of the third image, a flash duration parameter, a flash intensity parameter, a flash color temperature parameter, or a flash directionality parameter.

5. The method of claim 1 , wherein determining the one or more illumination control parameters for the illumination component for the third image includes:
determining the one or more illumination control parameters for the illumination component for the third image such that the third image satisfies an illumination criterion, wherein the third image satisfies the illumination criterion when a luminosity histogram of the third image indicates greater illuminance distribution compared to the delta luminosity histogram.

6. The method of claim 1 , wherein generating the foreground estimation includes normalizing the first image and the second image based on camera settings.

7. The method of claim 1 , wherein generating the foreground estimation includes aligning the first and second images.

8. The method of claim 1 , wherein generating the foreground estimation includes removing a clipped portion that corresponds to the first image and not the second image.

9. The method of claim 1 , wherein generating the foreground estimation includes blurring the first and second images according to predefined blur criteria.

10. The method of claim 1 , wherein generating the foreground estimation of the scene includes generating the foreground estimation of the scene based at least in part on a selected focus area and the comparison of the second image and the first image.

11. The method of claim 1 , wherein determining the one or more illumination control parameters for the third image of the scene that satisfy a foreground-background balance criterion based on the delta luminosity histogram includes determining the one or more illumination control parameters based on pixels within a portion of the delta luminosity histogram.

12. The method of claim 11 , wherein determining the one or more illumination control parameters based on pixels within the portion of the delta luminosity histogram includes:
determining a luminance gain value based on pixels within the portion of the delta luminosity histogram; and
determining the one or more illumination control parameters based on the luminance gain value.

13. An electronic device comprising:
an image sensor;
an illumination component;
a non-transitory memory;
one or more processors; and
one or more programs, wherein the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, the one or more programs including instructions for:
obtaining, by the image sensor, a first image of a scene while the illumination component is set to an inactive state;
obtaining, by the image sensor, a second image of the scene while the illumination component is set to a pre-flash state;
generating a foreground estimation of the scene based on relative brightness differences of pixels of the first image and pixels of the second image;
generating a delta luminosity histogram based on the foreground estimation;
determining one or more illumination control parameters for the illumination component for a third image of the scene that satisfy a foreground-background balance criterion based on the delta luminosity histogram; and
obtaining, by the image sensor, the third image of the scene while the illumination component is set to an active state in accordance with the one or more illumination control parameters.

14. The electronic device of claim 13 , wherein the one or more illumination control parameters set a flash period of the illumination component for the third image relative to a predefined exposure period of the third image.

15. The electronic device of claim 13 , wherein the one or more programs include instructions for determining one or more image parameters for the image sensor for the third image based on the foreground estimation in order to discriminate the foreground data from the background data within the scene, wherein one of the one or more image parameters sets an exposure period for the third image, and
wherein the one or more illumination control parameters set a flash period of the illumination component for the third image relative to the exposure period of the third image.

16. The electronic device of claim 13 , wherein the one or more illumination control parameters correspond to one of a flash timing parameter relative to an exposure of the third image, a flash duration parameter, a flash intensity parameter, a flash color temperature parameter, or a flash directionality parameter.

17. The electronic device of claim 13 , wherein determining the one or more illumination control parameters for the illumination component for the third image includes:
determining the one or more illumination control parameters for the illumination component for the third image such that the third image satisfies an illumination criterion, wherein the third image satisfies the illumination criterion when a luminosity histogram of the third image indicates greater illuminance distribution compared to the delta luminosity histogram.

18. The electronic device of claim 13 , wherein determining the one or more illumination control parameters for the third image of the scene that satisfy a foreground-background balance criterion based on the delta luminosity histogram includes determining the one or more illumination control parameters based on pixels within a portion of the delta luminosity histogram.

19. The electronic device of claim 18 , wherein determining the one or more illumination control parameters base

Assignees

• Apple Inc

Inventors

• Gatt Alexis
• Sachs Todd S
• Johnson Garrett M
• Moelgaard Claus

Classifications

• H04N23/73

  by influencing the exposure time · CPC title

• H04N23/74 Primary

  by influencing the scene brightness using illuminating means · CPC title

• H04N23/71

  Circuitry for evaluating the brightness variation · CPC title

• G06T7/11 Primary

  Region-based segmentation · CPC title

• G06T7/174

  involving the use of two or more images · CPC title