Low Light Video Image Stabilization Strength Modulation

Patent metadata

Field	Value
Publication number	US-2016006935-A1
Application number	US-201414467998-A
Country	US
Kind code	A1
Filing date	Aug 25, 2014
Priority date	Jul 6, 2014
Publication date	Jan 7, 2016
Grant date	—

Abstract

Techniques to improve a digital image capture device's ability to stabilize a video stream in low light capture conditions are presented. In general, techniques are disclosed for stabilizing video images using an overscan region and a look-ahead technique enabled by buffering a number of video input frames before generating a first stabilized video output frame. More particularly, techniques are disclosed for buffering an initial number of input frames so that a “current” frame can use time stamped image capture device motion data from both “past” and “future” frames to adjust the strength of a stabilization metric value so as to keep the current frame within its overscan. The calculated adjustments to the strength of the stabilization metric value for a particular frame may be limited to a target max strength value that is determined based, at least in part, on an estimated amount of motion blur in the current frame.

First claim

1 . A non-transitory program storage device comprising instructions stored thereon to cause one or more processors to:
capture a first time-sequenced set of images, wherein each image in the first time-sequenced set of images is associated with a capture time and motion information; select a current image from the first time-sequenced set of images; identify a first stabilization metric value for the current image; adjust the first stabilization metric value based, at least in part, on an estimated amount of motion blur in the current image, wherein the estimated amount of motion blur in the current image is based, at least in part, on at least part of the motion information associated with the current image; apply the adjusted first stabilization metric value to the current image to stabilize the current image; and store the stabilized current image in a memory.

2 . The non-transitory program storage device of claim 1 , wherein the motion information for each image comprises gyroscope information.

3 . The non-transitory program storage device of claim 2 , wherein the gyroscope information comprises one or more time stamped gyroscope data points.

4 . The non-transitory program storage device of claim 1 , wherein the instructions to estimate the amount of motion blur in the current image based, at least in part, on at least part of the motion information associated with the current image further comprise instructions to calculate an image row blur metric for one or more rows of pixels in the current image.

5 . The non-transitory program storage device of claim 4 , wherein the instructions to estimate the amount of motion blur in the current image based, at least in part, on at least part of the motion information associated with the current image further comprise instructions to calculate an overall image blur metric based, at least in part, on the one or more calculated image row blur metrics for the current image.

6 . The non-transitory program storage device of claim 3 , wherein the instructions to estimate the amount of motion blur in the current image based, at least in part, on at least part of the motion information associated with the current image further comprise instructions to calculate the beginning and ending integration time of one or more rows of pixels in the current image based, at least in part, on one or more of: one or more time stamped gyroscope data points for the current image, a rolling shutter readout time, and an exposure time.

7 . The non-transitory program storage device of claim 1 , wherein the instructions to adjust the first stabilization metric value based, at least in part, on an estimated amount of motion blur in the current image further comprise instructions to limit the first stabilization metric value to a calculated target maximum value.

8 . A video image stabilization method, comprising:
capturing a first time-sequenced set of images, wherein each image in the first time-sequenced set of images is associated with a capture time and motion information; selecting a current image from the first time-sequenced set of images; identifying a first stabilization metric value for the current image; adjusting the first stabilization metric value based, at least in part, on an estimated amount of motion blur in the current image, wherein the estimated amount of motion blur in the current image is based, at least in part, on at least part of the motion information associated with the current image; applying the adjusted first stabilization metric value to the current image to stabilize the current image; and storing the stabilized current image in a memory.

9 . The method of claim 8 , wherein the motion information for each image comprises one or more time stamped gyroscope data points.

10 . The method of claim 8 , wherein the act of estimating the amount of motion blur in the current image based, at least in part, on at least part of the motion information associated with the current image further comprises calculating an image row blur metric for one or more rows of pixels in the current image.

11 . The method of claim 10 , wherein the act of estimating the amount of motion blur in the current image based, at least in part, on at least part of the motion information associated with the current image further comprises calculating an overall image blur metric based, at least in part, on the one or more calculated image row blur metrics for the current image.

12 . The method of claim 9 , wherein the act of estimating the amount of motion blur in the current image based, at least in part, on at least part of the motion information associated with the current image further comprises calculating the beginning and ending integration time of one or more rows of pixels in the current image based, at least in part, on one or more of: one or more time stamped gyroscope data points for the current image, a rolling shutter readout time, and an exposure time.

13 . The method of claim 8 , wherein the act of adjusting the first stabilization metric value based, at least in part, on an estimated amount of motion blur in the current image further comprises limiting the first stabilization metric value to a calculated target maximum value.

14 . A video image capture device, comprising:
an image sensor; memory communicatively coupled to the image sensor; one or more motion sensors; one or more processors operatively coupled to the image sensor, the memory and the one or more motion sensors configured to:
capture a first time-sequenced set of images, wherein each image in the first time-sequenced set of images is associated with a capture time and motion information;
select a current image from the first time-sequenced set of images;
identify a first stabilization metric value for the current image;
adjust the first stabilization metric value based, at least in part, on an estimated amount of motion blur in the current image, wherein the estimated amount of motion blur in the current image is based, at least in part, on at least part of the motion information associated with the current image;
apply the adjusted first stabilization metric value to the current image to stabilize the current image; and
store the stabilized current image in a memory.

15 . The video image capture device of claim 14 , wherein the motion information for each image comprises gyroscope information.

16 . The video image capture device of claim 15 , wherein the gyroscope information comprises one or more time stamped gyroscope data points.

17 . The video image capture device of claim 14 , wherein the instructions to estimate the amount of motion blur in the current image based, at least in part, on at least part of the motion information associated with the current image further comprise instructions to calculate an image row blur metric for one or more rows of pixels in the current image.

18 . The video image capture device of claim 17 , wherein the instructions to estimate the amount of motion blur in the current image based, at least in part, on at least part of the motion information associated with the current image further comprise instructions to calculate an overall image blur metric based, at least in part, on the one or more calculated image row blur metrics for the current image.

19 . The video image capture device of claim 16 , wherein the instructions to estimate the amount of motion blur in the current image based, at least in part, on at least part of the motion information associated with the curren

Assignees

• Apple Inc

Inventors

• Zhou Jianping
• Beysserie Sebastien X

Classifications

• H04N23/683

  performed by a processor, e.g. controlling the readout of an image memory · CPC title

• H04N23/6812

  based on additional sensors, e.g. acceleration sensors · CPC title

• H04N23/6811 Primary

  based on the image signal · CPC title

• H04N5/23254 Primary

  Electricity · mapped topic