Image alignment for computational photography

Patent metadata

Field	Value
Publication number	US-11711613-B2
Application number	US-202117241742-A
Country	US
Kind code	B2
Filing date	Apr 27, 2021
Priority date	Apr 27, 2021
Publication date	Jul 25, 2023
Grant date	Jul 25, 2023

Abstract

Image frames for computational photography may be corrected, such as through rolling shutter correction (RSC), prior to fusion of the image frames to reduce wobble and jitter artifacts present in a video sequence of HDR-enhanced image frames. First and second motion data regarding motion of the image capture device may be determined for times corresponding to the capturing of the first and second image frames, respectively. The rolling shutter correction (RSC) may be applied to the first and second image frames based on both the first and second motion data. The corrected first and second image frames may then be aligned and fused to obtain a single output image frame with higher dynamic range than either of the first or second image frames.

First claim

What is claimed is:

1. A method, comprising:
receiving a first image frame obtained at a first time and a second image frame obtained at a second time later than the first time;
determining a corrected first image frame based on the first image frame, first motion data corresponding to motion at the first time when obtaining the first image frame, and second motion data corresponding to motion at the second time when obtaining the second image frame;
determining a corrected second image frame based on the second image frame, the first motion data, and the second motion data;
determining an alignment correction to align the corrected first image frame to the corrected second image frame, wherein the alignment correction is based on the first motion data and the second motion data when a distance difference between foreground and background objects in the first image is not below a first depth threshold, and wherein the alignment correction is based on a matched feature distance between matched features between the first image frame and the second image frame when the distance difference between foreground and background objects in the first image is below the first depth threshold; and
determining a fused image frame based on the first image frame, the second image frame, and the alignment correction.

2. The method of claim 1 , wherein determining a corrected first image frame comprises applying a first rolling shutter correction (RSC) to the first image frame, and wherein determining a corrected second image frame comprises applying a second rolling shutter correction (RSC) to the second image frame, the first rolling shutter correction (RSC) and the second rolling shutter correction (RSC) based on both the first motion data and the second motion data.

3. The method of claim 1 , wherein the first motion data and the second motion data comprise gyroscope data.

4. The method of claim 1 , wherein determining an alignment correction comprises:
determining the matched features between the first image frame and the second image frame.

5. The method of claim 1 , wherein:
the first image frame is captured using a first exposure time, and
the second image frame is captured using a second exposure time, the second exposure time being longer than the first exposure time.

6. The method of claim 1 , wherein determining an alignment correction comprises:
determining a first direction of global motion; and
aligning the corrected first image frame to the corrected second image frame in a second direction opposite that of the first direction.

7. A device, comprising:
a processor; and
a memory coupled to the processor and storing instructions that, when executed by the processor, cause the device to perform operations comprising:
receiving a first image frame obtained at a first time and a second image frame obtained at a second time later than the first time;
determining a corrected first image frame based on the first image frame, first motion data corresponding to motion at the first time when obtaining the first image frame, and second motion data corresponding to motion at the second time when obtaining the second image frame;
determining a corrected second image frame based on the second image frame, the first motion data, and the second motion data;
determining an alignment correction to align the corrected first image frame to the corrected second image frame, wherein the alignment correction is based on the first motion data and the second motion data when a distance difference between foreground and background objects in the first image is not below a first depth threshold, and wherein the alignment correction is based on a matched feature distance between matched features between the first image frame and the second image frame when the distance difference between foreground and background objects in the first image is below the first depth threshold; and
determining a fused image frame based on the first image frame, the second image frame, and the alignment correction.

8. The device of claim 7 , wherein determining a corrected first image frame comprises applying a first rolling shutter correction (RSC) to the first image frame, and wherein determining a corrected second image frame comprises applying a second rolling shutter correction (RSC) to the second image frame, the first rolling shutter correction (RSC) and the second rolling shutter correction (RSC) based on both the first motion data and the second motion data.

9. The device of claim 7 , wherein the first motion data and the second motion data comprise gyroscope data.

10. The device of claim 7 , wherein determining an alignment correction comprises:
determining the matched features between the first image frame and the second image frame.

11. The device of claim 7 , wherein:
the first image frame is captured using a first exposure time, and
the second image frame is captured using a second exposure time, the second exposure time being longer than the first exposure time.

12. The device of claim 7 , wherein determining an alignment correction comprises:
determining a first direction of global motion; and
aligning the corrected first image frame to the corrected second image frame in a second direction opposite that of the first direction.

13. A non-transitory computer-readable medium storing instructions that, when executed by a processor of a device, cause the device to perform operations comprising:
receiving a first image frame obtained at a first time and a second image frame obtained at a second time later than the first time;
determining a corrected first image frame based on the first image frame, first motion data corresponding to motion at the first time when obtaining the first image frame, and second motion data corresponding to motion at the second time when obtaining the second image frame, wherein the alignment correction is based on the first motion data and the second motion data when a distance difference between foreground and background objects in the first image is not below a first depth threshold, and wherein the alignment correction is based on a matched feature distance between matched features between the first image frame and the second image frame when the distance difference between foreground and background objects in the first image is below the first depth threshold;
determining a corrected second image frame based on the second image frame, the first motion data, and the second motion data;
determining an alignment correction to align the corrected first image frame to the corrected second image frame; and
determining a fused image frame based on the first image frame, the second image frame, and the alignment correction.

14. The non-transitory computer-readable medium of claim 13 , wherein determining a corrected first image frame comprises applying a first rolling shutter correction (RSC) to the first image frame, and wherein determining a corrected second image frame comprises applying a second rolling shutter correction (RSC) to the second image frame, the first rolling shutter correction (RSC) and the second rolling shutter correction (RSC) based on both the first motion data and the second motion data.

15. The non-transitory computer-readable medium of claim 13 , wherein the first motion data and the second motion data comprise gyroscope data.

16. The non-transitory computer-readable medium of claim 13 , wherein determining an alignment correction comprises:
determining the matched features between the first image frame and the second image frame.

17.

Assignees

• Qualcomm Inc

Inventors

• Chuang Shang-Chih
• Gaizman Ron
• Veeramalla Pradeep
• Liu Shizhong
• Liu Weiliang
• Jiang Xiaoyun
• Ravirala Narayana Karthik
• Luo Jiafu
• Anekere Narayan Srinath
• Zou Wei

Classifications

• H04N23/689 Primary

  Motion occurring during a rolling shutter mode · CPC title

• H04N23/6812

  based on additional sensors, e.g. acceleration sensors · CPC title

• G06T5/92

  based on global image properties · CPC title

• G06T5/50 Primary

  using two or more images, e.g. averaging or subtraction · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title