Techniques for correcting images in flash photography

Patent metadata

Field	Value
Publication number	US-12356083-B2
Application number	US-202217950859-A
Country	US
Kind code	B2
Filing date	Sep 22, 2022
Priority date	Sep 22, 2022
Publication date	Jul 8, 2025
Grant date	Jul 8, 2025

Abstract

Embodiments disclosed herein are directed to devices and methods for performing spatially-varying brightness correction of a flash image. The spatially-varying brightness correction utilizes a set of images that includes one or more images captured under flash illumination and one or more images captured without flash illumination, which are used to decompose one of the flash images or an image generated therefrom into a flash contribution image and an ambient contribution image. The flash contribution image and the ambient contribution image are recombined to generate a corrected flash image, and information about the flash module and/or scene is used to adjust the relative contributions of these images during recombination.

First claim

What is claimed is:

1. An image processing method, comprising:
obtaining a set of images from a set of cameras, the set of images comprising a set of flash images;
generating a fused flash image from two or more flash images of the set of flash images;
decomposing the fused flash image using a reference image to generate a flash contribution image and an ambient contribution image;
performing a spatially-varying brightness adjustment to the flash contribution image; and
recombining the brightness-adjusted flash contribution image and the ambient contribution image to generate a corrected flash image.

2. The method of claim 1 , comprising:
selecting a source image; and
using a style transfer operation to generate an output image from the corrected flash image and the source image.

3. The method of claim 2 , wherein:
the source image is the fused flash image.

4. The method of claim 1 , wherein:
the set of images comprises a set of ambient images; and
the reference image is selected or generated from the set of ambient images.

5. The method of claim 1 , wherein:
the set of images comprises a set of pre-flash images; and
the reference image is selected or generated from the set of pre-flash images.

6. The method of claim 1 , wherein:
performing the spatially-varying brightness adjustment comprises spatially adjusting the brightness of the flash contribution image using a depth map.

7. The method of claim 1 , wherein:
performing the spatially-varying brightness adjustment comprises spatially adjusting the brightness of the flash contribution image using a reflectivity map.

8. The method of claim 1 , wherein:
performing the spatially-varying brightness adjustment comprises spatially adjusting the brightness of the flash contribution image using a flash illumination profile.

9. The method of claim 1 , comprising:
performing a first white balancing operation to color correct the flash contribution image; and
performing a second white balancing operation to color correct the ambient contribution image.

10. The method of claim 9 , wherein:
the second white balancing operation uses the color-corrected flash contribution image to generate a color correction map.

11. The method of claim 1 , wherein:
performing the spatially-varying brightness adjustment comprises spatially adjusting the brightness of the flash contribution image using contextual information about a scene captured by the set of images.

12. A device comprising:
a set of cameras;
a flash module;
a memory; and
one or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions causing the one or more processors to:
obtain a set of images from the set of cameras, the set of images comprising a set of flash images;
obtain a depth map;
select or generate a reference image from the set of images;
select or generate an input image from the set of flash images;
decompose the input image using the reference image to generate a flash contribution image and an ambient contribution image;
perform a spatially-varying brightness adjustment to the flash contribution image using the depth map; and
recombine the brightness-adjusted flash contribution image and the ambient contribution image to generate a corrected flash image.

13. The device of claim 12 , comprising a depth sensor, wherein:
obtaining the depth map comprises deriving the depth map from depth information generated by the depth sensor.

14. The device of claim 12 , wherein:
performing the spatially-varying brightness adjustment comprises performing the spatially-varying brightness adjustment using a flash illumination profile associated with the flash module.

15. The device of claim 14 , wherein:
the flash illumination profile is selected from a set of candidate flash illumination profiles.

16. The device of claim 12 , wherein:
performing the spatially-varying brightness adjustment comprises performing the spatially-varying brightness adjustment using contextual information about a scene captured by the set of images.

17. The device of claim 12 , wherein:
the set of images comprises a set of ambient images; and
the reference image is selected or generated from the set of ambient images.

18. A device comprising:
a set of cameras;
a flash module;
a memory; and
one or more processors operatively coupled to the memory, wherein the one or more processors are configured to execute instructions causing the one or more processors to:
obtain a set of images from the set of cameras, the set of images comprising a set of flash images;
select or generate an input image from the set of flash images;
select or generate a reference image from the set of images;
decompose the input image using the reference image to generate a flash contribution image and an ambient contribution image;
perform a spatially-varying brightness adjustment to the flash contribution image using a reflectivity map; and
recombine the brightness-adjusted flash contribution image and the ambient contribution image to generate a corrected flash image.

19. The device of claim 18 , wherein the one or more processors are configured to execute the instructions causing the one or more processors to:
obtain a depth map; and
generate the reflectivity map using the flash contribution image and the depth map.

20. The device of claim 19 , wherein:
generating the reflectivity map comprises generating the reflectivity map using a flash illumination profile associated with the flash module.

21. The device of claim 18 , wherein:
the input image is a fused flash image generated from two or more flash images of the set of flash images.

22. The device of claim 18 , wherein the one or more processors are configured to execute the instructions causing the one or more processors to:
down sample the input image.

Assignees

• Apple Inc

Inventors

• Zhang Bosheng
• Baldauf Tobias
• Romanenko Ilya
• Bonnier Nicolas P
• Bedard Noah D
• Ludlow Alexandra S
• Finlayson Graham D
• De Villiers Jason P
• Moelgaard Claus
• Hung Hau Ling
• Hubel Paul M

Classifications

• G06T7/50

  Depth or shape recovery · CPC title

• H04N9/73

  Colour balance circuits, e.g. white balance circuits or colour temperature control (camera processing pipelines for colour balance H04N23/88) · CPC title

• G06T2207/20221

  Image fusion; Image merging · CPC title

• G06T5/50

  using two or more images, e.g. averaging or subtraction · CPC title

• H04N23/741

  by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors · CPC title