Video image stabilization with enforced stabilization constraints

Patent metadata

Field	Value
Publication number	US-9787902-B1
Application number	US-201615179680-A
Country	US
Kind code	B1
Filing date	Jun 10, 2016
Priority date	Jun 10, 2016
Publication date	Oct 10, 2017
Grant date	Oct 10, 2017

Abstract

Techniques to improve a digital image capture device's ability to stabilize a video stream—while enforcing desired stabilization constraints on particular images in the video stream—are presented that utilize an overscan region and a look-ahead technique enabled by buffering a number of video input frames before generating a first stabilized video output frame. More particularly, techniques are disclosed for buffering an initial number of input frames so that a “current” frame can use motion data from both “past” and “future” frames to adjust the value of a stabilization strength parameter and/or the weighted contribution of particular frames from the buffer in the determination of stabilization motion values for the current frame. Such techniques keep the current frame within its overscan and ensure that the stabilization constraints are enforced, while maintaining desired smoothness in the video stream. In some embodiments, the stabilization constraint may comprise a maximum allowed frame displacement.

First claim

The invention claimed is:

1. A non-transitory program storage device comprising instructions stored thereon to cause one or more processors to:
obtain a first time-sequenced set of images, wherein:
each image in the first time-sequenced set of images has a capture time and motion data;
one image in the first time-sequenced set of images is designated as a current image;
a first one or more images within the first time-sequenced set of images having capture times before the current image are designated past images; and
a second one or more images within the first time-sequenced set of images having capture times after the current image are designated future images;

select a first image from the first time-sequenced set of images to meet a first stabilization constraint;
identify a weighting value for each image in the first time-sequenced set of images based, at least in part, on the proximity of the respective image's capture time to the capture time of the current image;
adjust the identified weighting value for the selected first image according to a first weighting adjustment value;
identify a first stabilization motion value for the current image in the first time-sequenced set of images based, at least in part, on the identified weighting values and the adjusted weighting value for the selected first image;
apply the first stabilization motion value to the current image to stabilize the current image; and
store the stabilized current image in a memory.

2. The non-transitory program storage device of claim 1 , wherein the motion data for each image comprises gyroscope information.

3. The non-transitory program storage device of claim 1 , wherein the identified weighting value for each image in the first time-sequenced set of images is further based, at least in part, on a stabilization strength parameter value for the current image.

4. The non-transitory program storage device of claim 1 , wherein the first weighting adjustment value is further determined based, at least in part, on the proximity of the selected first image's capture time to the capture time of the current image.

5. The non-transitory program storage device of claim 1 , wherein the first weighting adjustment value is further determined based, at least in part, on whether the selected first image is presently a past image or a future image.

6. The non-transitory program storage device of claim 1 , wherein the first stabilization constraint comprises a maximum allowed displacement for the stabilization of the selected first image.

7. The non-transitory program storage device of claim 1 , wherein the instructions to apply the first stabilization motion value to the current image to stabilize the current image further comprise instructions to aggregate one or more pixels with the current image, so as to maintain a consistent Field of View (FOV) between the current image and the other images in the first time-sequenced set of images.

8. A video image stabilization method, comprising:
obtaining a first time-sequenced set of images, wherein:
each image in the first time-sequenced set of images has a capture time and motion data;
one image in the first time-sequenced set of images is designated as a current image;
a first one or more images within the first time-sequenced set of images having capture times before the current image are designated past images; and
a second one or more images within the first time-sequenced set of images having capture times after the current image are designated future images;

selecting a first image from the first time-sequenced set of images to meet a first stabilization constraint;
identifying a weighting value for each image in the first time-sequenced set of images based, at least in part, on the proximity of the respective image's capture time to the capture time of the current image;
adjusting the identified weighting value for the selected first image according to a first weighting adjustment value;
identifying a first stabilization motion value for the current image in the first time-sequenced set of images based, at least in part, on the identified weighting values and the adjusted weighting value for the selected first image;
applying the first stabilization motion value to the current image to stabilize the current image; and
storing the stabilized current image in a memory.

9. The method of claim 8 , wherein the motion data for each image comprises one or more time stamped gyroscope data points.

10. The method of claim 8 , wherein the identified weighting value for each image in the first time-sequenced set of images is further based, at least in part, on a stabilization strength parameter value for the current image.

11. The method of claim 8 , wherein the first weighting adjustment value is further determined based, at least in part, on the proximity of the selected first image's capture time to the capture time of the current image.

12. The method of claim 8 , wherein the first stabilization constraint comprises a maximum allowed displacement for the stabilization of the selected first image.

13. The method of claim 8 , wherein the act of applying the first stabilization motion value to the current image to stabilize the current image further comprises aggregating one or more pixels with the current image, so as to maintain a consistent FOV between the current image and the other images in the first time-sequenced set of images.

14. A video image capture device, comprising:
an image sensor;
memory communicatively coupled to the image sensor;
one or more motion sensors;
one or more processors operatively coupled to the image sensor, the memory and the one or more motion sensors configured to execute instructions causing the one or more processors to:
obtain a first time-sequenced set of images, wherein:
each image in the first time-sequenced set of images has a capture time and motion data;
one image in the first time-sequenced set of images is designated as a current image;
a first one or more images within the first time-sequenced set of images having capture times before the current image are designated past images; and
a second one or more images within the first time-sequenced set of images having capture times after the current image are designated future images;

select a first image from the first time-sequenced set of images to meet a first stabilization constraint;
identify a weighting value for each image in the first time-sequenced set of images based, at least in part, on the proximity of the respective image's capture time to the capture time of the current image;
adjust the identified weighting value for the selected first image according to a first weighting adjustment value;
identify a first stabilization motion value for the current image in the first time-sequenced set of images based, at least in part, on the identified weighting values and the adjusted weighting value for the selected first image;
apply the first stabilization motion value to the current image to stabilize the current image; and
store the stabilized current image in a memory.

15. The video image capture device of claim 14 , wherein the motion data for each image comprises gyroscope information.

16. The video image capture device of claim 14 , wherein the identified weighting value for each image in the first time-sequenced set of images is further based, at least in part, on a stabilization strength parameter value for the current image.

17. The video image capture device of claim 14 , wherein the first weighting adjustment value is

Assignees

• Apple Inc

Inventors

• Beysserie Sebastien X
• Zhou Jianping
• Ben Soussan Stephane S

Classifications

• H04N23/6812 Primary

  based on additional sensors, e.g. acceleration sensors · CPC title

• H04N23/683 Primary

  performed by a processor, e.g. controlling the readout of an image memory · CPC title

• H04N5/23258

  Electricity · mapped topic

• H04N5/23267 Primary

  Electricity · mapped topic