Methods and systems for shift estimation for one or more output frames

Patent metadata

Field	Value
Publication number	US-11825207-B1
Application number	US-202217734950-A
Country	US
Kind code	B1
Filing date	May 2, 2022
Priority date	May 2, 2022
Publication date	Nov 21, 2023
Grant date	Nov 21, 2023

Abstract

The present disclosure generally relates to exposure control for image capture devices. One example method generally includes receiving sensor data indicating a movement associated with an image capture device capturing a first frame and a second frame, wherein the first frame is associated with a higher exposure setting as compared to the second frame; predicting, based on the sensor data, a shift for aligning one or more features of the first frame and one or more features of the second frame; applying the shift to align the one or more features of the first frame and the one or more features of second frame; and generating a frame output at least in part by combining at least a portion of the first frame with at least a portion of the second frame.

First claim

What is claimed is:

1. An apparatus for frame alignment, comprising:
at least one memory; and
at least one processor coupled to the at least one memory and configured to:
receive sensor data indicating a movement associated with an image capture device capturing a first frame and a second frame, wherein the first frame is associated with a higher exposure setting as compared to the second frame;
predict, based on the sensor data and an estimate of a delay associated with capturing at least one high exposure setting frame and at least one low exposure setting frame to be combined for generating at least one high dynamic range (HDR) output, a shift for aligning one or more features of the first frame and one or more features of the second frame;
apply the shift to align the one or more features of the first frame and the one or more features of second frame; and
generate a frame output at least in part by combining at least a portion of the first frame with at least a portion of the second frame.

2. The apparatus of claim 1 , wherein at least one processor is configured to predict the shift prior to the first frame and the second frame being captured.

3. The apparatus of claim 1 , wherein the at least one processor is configured to:
receive the estimate of the delay.

4. The apparatus of claim 1 , wherein the first frame being associated with a higher exposure setting as compared to the second frame comprises the first frame being associated with a longer exposure time as compared to the second frame.

5. The apparatus of claim 1 , wherein the at least one processor is configured to predict the shift based on a sensor-based alignment library generated using a trained machine learning model.

6. The apparatus of claim 5 , wherein the trained machine learning model comprises a recurring neural network.

7. The apparatus of claim 1 , wherein, to apply the shift, the at least one processor is configured to apply the shift to the first frame having the higher exposure setting to align the one or more features of the first frame with the one or more features of the second frame.

8. The apparatus of claim 1 , wherein the frame output comprises an HDR frame output.

9. The apparatus of claim 1 , wherein the at least one processor is configured to:
accumulate first image sensor data associated with the first frame in a first buffer;
accumulate second image sensor data associated with the second frame in a second buffer; and
apply the shift based on the first image sensor data in the first buffer while the first image sensor data is being accumulated.

10. The apparatus of claim 9 , wherein the at least one processor is configured to accumulate at least one of the first image sensor data or the second image sensor data from a center position of the first buffer or the second buffer, respectively.

11. The apparatus of claim 1 , wherein the sensor data comprises gyroscope data.

12. A method for frame alignment, comprising:
receiving sensor data indicating a movement associated with an image capture device capturing a first frame and a second frame, wherein the first frame is associated with a higher exposure setting as compared to the second frame;
predicting, based on the sensor data and an estimate of a delay associated with capturing at least one high exposure setting frame and at least one low exposure setting frame to be combined for generating at least one high dynamic range (HDR) output, a shift for aligning one or more features of the first frame and one or more features of the second frame;
applying the shift to align the one or more features of the first frame and the one or more features of second frame; and
generating a frame output at least in part by combining at least a portion of the first frame with at least a portion of the second frame.

13. The method of claim 12 , wherein the shift is predicted prior to the first frame and the second frame being captured.

14. The method of claim 12 , further comprising:
receiving the estimate of the delay.

15. The method of claim 12 , wherein the first frame being associated with a higher exposure setting as compared to the second frame comprises the first frame being associated with a longer exposure time as compared to the second frame.

16. The method of claim 12 , further comprising predicting the shift based on a sensor-based alignment library generated using a trained machine learning model.

17. The method of claim 16 , wherein the trained machine learning model comprises a recurring neural network.

18. The method of claim 12 , wherein applying the shift comprises applying the shift to the first frame having the higher exposure setting to align the one or more features of the first frame with the one or more features of the second frame.

19. The method of claim 12 , wherein the frame output comprises an HDR frame output.

20. The method of claim 12 , further comprising:
accumulating first image sensor data associated with the first frame in a first buffer;
accumulating second image sensor data associated with the second frame in a second buffer; and
applying the shift based on the first image sensor data in the first buffer while the first image sensor data is being accumulated.

21. The method of claim 20 , wherein at least one of the first image sensor data or the second image sensor data is accumulated from a center position of the first buffer or the second buffer, respectively.

22. The method of claim 12 , wherein the sensor data comprises gyroscope data.

23. A non-transitory computer-readable medium having instructions, which when executed by one or more processors, cause the one or more processors to:
receive sensor data indicating a movement associated with an image capture device capturing a first frame and a second frame, wherein the first frame is associated with a higher exposure setting as compared to the second frame;
predict, based on the sensor data and an estimate of a delay associated with capturing at least one high exposure setting frame and at least one low exposure setting frame to be combined for generating at least one high dynamic range (HDR) output, a shift for aligning one or more features of the first frame and one or more features of the second frame;
apply the shift to align the one or more features of the first frame and the one or more features of second frame; and
generate a frame output at least in part by combining at least a portion of the first frame with at least a portion of the second frame.

Assignees

• Qualcomm Inc

Inventors

• Ravirala Narayana Karthik
• Raj Chanchal
• Liu Shizhong
• Liu Weiliang
• Gaizman Ron
• Shi Lianming

Classifications

• H04N23/73 Primary

  by influencing the exposure time · CPC title

• H04N5/144

  Movement detection (for video coding H04N19/503; analysis of motion in general G06T7/20) · CPC title

• H04N23/741

  by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors · CPC title

• G06T2207/10016

  Video; Image sequence · CPC title

• G06T2207/10152

  Varying illumination · CPC title