System and method for flicker avoidance in multi-frame multi-exposure image captures

Patent metadata

Field	Value
Publication number	US-12231780-B2
Application number	US-202218059822-A
Country	US
Kind code	B2
Filing date	Nov 29, 2022
Priority date	Feb 9, 2022
Publication date	Feb 18, 2025
Grant date	Feb 18, 2025

Abstract

A method includes obtaining multiple image frames. The method also includes selecting, using at least one processing device of an electronic device, an asymmetrical image pair from the multiple image frames. The asymmetrical image pair includes a first image frame and a second image frame, where the first image frame has a shorter exposure than the second image frame. The method further includes identifying, using the at least one processing device, one or more features based on the asymmetrical image pair. The method also includes determining, using the at least one processing device, whether the first image frame contains flicker based on the one or more features. In addition, the method includes enabling or disabling, using the at least one processing device, the first image frame as a reference candidate based on the determination whether the first image frame contains flicker.

First claim

What is claimed is:

1. A method comprising:
obtaining multiple image frames;
selecting, using at least one processing device of an electronic device, an asymmetrical image pair from the multiple image frames, the asymmetrical image pair comprising a first image frame and a second image frame, the first image frame having a shorter exposure than the second image frame;
identifying, using the at least one processing device, one or more features based on the asymmetrical image pair, wherein identifying the one or more features based on the asymmetrical image pair comprises:
generating a difference image based on the first image frame and the second image frame; and
dividing the difference image into patches;

determining, using the at least one processing device, whether the first image frame contains flicker based on the one or more features; and
enabling or disabling, using the at least one processing device, the first image frame as a reference candidate based on the determination whether the first image frame contains flicker.

2. The method of claim 1 , wherein identifying the one or more features based on the asymmetrical image pair further comprises, for each patch of the difference image:
identifying a signal representing an average brightness across the patch; and
identifying a frequency signal representative of frequency components in the signal representing the average brightness across the patch.

3. The method of claim 2 , wherein determining whether the first image frame contains flicker comprises:
detecting a dominant frequency across each patch of the difference image based on the frequency signal for that patch;
searching the dominant frequencies across all patches for a largest normalized energy;
comparing the largest normalized energy to a threshold;
determining that the first image frame contains flicker when the largest normalized energy is greater than the threshold; and
determining that the first image frame does not contain flicker when the largest normalized energy is less than the threshold.

4. The method of claim 2 , wherein determining whether the first image frame contains flicker comprises:
detecting a dominant frequency across each patch of the difference image based on the frequency signal for that patch;
determining a weighted average of normalized energy across frequencies and patches;
comparing the weighted average of normalized energy to a threshold;
determining that the first image frame contains flicker when the weighted average of normalized energy is greater than the threshold; and
determining that the first image frame does not contain flicker when the weighted average of normalized energy is less than the threshold.

5. The method of claim 2 , wherein determining whether the first image frame contains flicker comprises:
detecting a dominant frequency across each patch of the difference image based on the frequency signal for that patch;
determining a number of patches for which the dominant frequency is greater than a first threshold;
determining that the first image frame contains flicker when the number of patches is greater than a second threshold; and
determining that the first image frame does not contain flicker when the number of patches is less than the second threshold.

6. The method of claim 2 , wherein:
determining whether the first image frame contains flicker comprises:
providing the frequency signals for the patches to a trained machine learning model; and
determining, using the trained machine learning model, whether the first image frame contains flicker; and

the trained machine learning model has been trained using multiple pairs of frequency signals, each pair of frequency signals including a flicker-free frequency signal and a corresponding frequency signal containing flicker.

7. The method of claim 1 , wherein:
the first image frame has an exposure value of EV-4 or EV-2; and
the second image frame has an exposure value of EV-0.

8. An electronic device comprising:
at least one processing device configured to:
obtain multiple image frames;
select an asymmetrical image pair from the multiple image frames, the asymmetrical image pair comprising a first image frame and a second image frame, the first image frame having a shorter exposure than the second image frame;
identify one or more features based on the asymmetrical image pair, wherein, to identify the one or more features based on the asymmetrical image pair, the at least one processing device is configured to:
generate a difference image based on the first image frame and the second image frame; and
divide the difference image into patches;

determine whether the first image frame contains flicker based on the one or more features; and
enable or disable the first image frame as a reference candidate based on the determination whether the first image frame contains flicker.

9. The electronic device of claim 8 , wherein, to identify the one or more features based on the asymmetrical image pair, the at least one processing device is further configured, for each patch of the difference image, to:
identify a signal representing an average brightness across the patch; and
identify a frequency signal representative of frequency components in the signal representing the average brightness across the patch.

10. The electronic device of claim 9 , wherein, to determine whether the first image frame contains flicker, the at least one processing device is configured to:
detect a dominant frequency across each patch of the difference image based on the frequency signal for that patch;
search the dominant frequencies across all patches for a largest normalized energy;
compare the largest normalized energy to a threshold;
determine that the first image frame contains flicker when the largest normalized energy is greater than the threshold; and
determine that the first image frame does not contain flicker when the largest normalized energy is less than the threshold.

11. The electronic device of claim 9 , wherein to determine whether the first image frame contains flicker, the at least one processing device is configured to:
detect a dominant frequency across each patch of the difference image based on the frequency signal for that patch;
determine a weighted average of normalized energy across frequencies and patches;
compare the weighted average of normalized energy to a threshold;
determine that the first image frame contains flicker when the weighted average of normalized energy is greater than the threshold; and
determine that the first image frame does not contain flicker when the weighted average of normalized energy is less than the threshold.

12. The electronic device of claim 9 , wherein, to determine whether the first image frame contains flicker, the at least one processing device is configured to:
detect a dominant frequency across each patch of the difference image based on the frequency signal for that patch;
determine a number of patches for which the dominant frequency is greater than a first threshold;
determine that the first image frame contains flicker when the number of patches is greater than a second threshold; and
determine that the first image frame does not contain flicker when the number of patches is less than the second threshold.

13. The electronic device of claim 9 , wherein:
to determine whether the first image frame contains flicker, the at least one processing device is configured to:
provide the frequency signals for the patches to a trained machine learning model; and
determine, using the trained machine learning model, whether the first im

Assignees

• Samsung Electronics Co Ltd

Inventors

• Le Nguyen Thang Long
• Nadir Zeeshan
• Glotzbach John W
• Sheikh Hamid R

Classifications

• G06V10/44

  Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components · CPC title

• G06T2207/10144

  Varying exposure · CPC title

• G06V10/761

  Proximity, similarity or dissimilarity measures · CPC title

• G06T2207/20224

  Image subtraction · CPC title

• G06T7/11

  Region-based segmentation · CPC title