Combined optical and electronic image stabilization

Patent metadata

Field	Value
Publication number	US-9979889-B2
Application number	US-201715457377-A
Country	US
Kind code	B2
Filing date	Mar 13, 2017
Priority date	Jun 7, 2015
Publication date	May 22, 2018
Grant date	May 22, 2018

Abstract

Techniques to improve a digital image capture device's ability to stabilize a video stream are presented. According to some embodiments, improved stabilization of captured video frames is provided by intelligently harnessing the complementary effects of both optical image stabilization (OIS) and electronic image stabilization (EIS). In particular, OIS may be used to remove intra-frame motion blur that is typically lower in amplitude and dominates with longer integration times, while EIS may be used to remove residual unwanted frame-to-frame motion that is typically larger in amplitude. The techniques disclosed herein may also leverage information provided from the image capture device's OIS system to perform improved motion blur-aware video stabilization strength modulation, which permits better video stabilization performance in low light conditions, where integration times tend to be longer, thus leading to a greater amount of motion blurring in the output stabilized video.

First claim

The invention claimed is:

1. A non-transitory program storage device comprising instructions stored thereon to cause one or more processors to:
obtain a first set of images and one or more motion information samples associated with the first set of images;
perform, during the obtaining of one or more of the first set of images, a filtering operation on at least one of the one or more motion information samples, wherein the filtering operation comprises identifying a first set of high frequency motion information data and a first set of low frequency motion information data;
cause a first amount of optical image stabilization (OIS) to be applied to one or more images in the first set of images based, at least in part, on the first set of high frequency motion information data;
cause a first amount of electronic image stabilization (EIS) to be applied to one or more images in the first set of images based, at least in part, on the first set of low frequency motion information data and the applied first amount of OIS; and
cause the stabilized first set of images to be stored in a memory.

2. The non-transitory program storage device of claim 1 , wherein a first image from the first set of images is associated with a first subset of the motion information samples.

3. The non-transitory program storage device of claim 2 , wherein a second image from the first set of images is associated with a second subset of the motion information samples that is different from the first subset of the motion information samples.

4. The non-transitory program storage device of claim 3 , wherein a second amount of OIS is applied to the first image, and wherein the second amount of OIS applied to the first image is based only on an identified first set of high frequency motion information data from the first subset of the motion information samples.

5. The non-transitory program storage device of claim 3 , wherein a second amount of EIS is applied to the first image, and wherein the second amount of EIS applied to the first image is based on at least an identified first set of low frequency motion information data from the first subset of the motion information samples and an identified first set of low frequency motion information data from the second subset of the motion information samples.

6. The non-transitory program storage device of claim 1 , wherein the filtering operation comprises use of a first cut-off frequency, wherein the first set of high frequency motion data comprises motion data having a frequency component greater than the first cut-off frequency, and wherein the first set of low frequency motion data comprises motion data having a frequency component less than the first cut-off frequency.

7. The non-transitory program storage device of claim 6 , wherein the first cut-off frequency comprises a movable cut-off frequency.

8. A video image stabilization method, comprising:
obtaining a first set of images and one or more motion information samples associated with the first set of images;
performing, during the obtaining of one or more of the first set of images, a filtering operation on at least one of the one or more motion information samples, wherein the filtering operation comprises identifying a first set of high frequency motion information data and a first set of low frequency motion information data;
causing a first amount of optical image stabilization (OIS) to be applied to one or more images in the first set of images based, at least in part, on the first set of high frequency motion information data;
causing a first amount of electronic image stabilization (EIS) to be applied to one or more images in the first set of images based, at least in part, on the first set of low frequency motion information data and the applied first amount of OIS; and
causing the stabilized first set of images to be stored in a memory.

9. The method of claim 8 , wherein a first image from the first set of images is associated with a first subset of the motion information samples.

10. The method of claim 9 , wherein a second image from the first set of images is associated with a second subset of the motion information samples that is different from the first subset of the motion information samples.

11. The method of claim 10 , wherein a second amount of OIS is applied to the first image, and wherein the second amount of OIS applied to the first image is based only on an identified first set of high frequency motion information data from the first subset of the motion information samples.

12. The method of claim 10 , wherein a second amount of EIS is applied to the first image, and wherein the second amount of EIS applied to the first image is based on at least an identified first set of low frequency motion information data from the first subset of the motion information samples and an identified first set of low frequency motion information data from the second subset of the motion information samples.

13. The method of claim 8 , wherein the filtering operation comprises use of a first cut-off frequency, wherein the first set of high frequency motion data comprises motion data having a frequency component greater than the first cut-off frequency, and wherein the first set of low frequency motion data comprises motion data having a frequency component less than the first cut-off frequency.

14. The method of claim 13 , wherein the first cut-off frequency comprises a movable cut-off frequency.

15. A video image capture device, comprising:
a camera module comprising an image sensor;
memory communicatively coupled to the image sensor;
one or more motion sensors;
a lens assembly mechanically coupled to the camera module, wherein the lens assembly comprises one or more parts subject to optical image stabilization (OIS);
one or more processors operatively coupled to the image sensor, the memory, and the one or more motion sensors configured to perform instructions to cause the one or more processors to:
obtain a first set of images and one or more motion information samples associated with the first set of images;
perform, during the obtaining of one or more of the first set of images, a filtering operation on at least one of the one or more motion information samples, wherein the filtering operation comprises identifying a first set of high frequency motion information data and a first set of low frequency motion information data;
cause a first amount of OIS to be applied to one or more images in the first set of images based, at least in part, on the first set of high frequency motion information data;
cause a first amount of electronic image stabilization (EIS) to be applied to one or more images in the first set of images based, at least in part, on the first set of low frequency motion information data and the applied first amount of OIS; and
cause the stabilized first set of images to be stored in a memory.

16. The device of claim 15 , wherein a first image from the first set of images is associated with a first subset of the motion information samples.

17. The device of claim 16 , wherein a second image from the first set of images is associated with a second subset of the motion information samples that is different from the first subset of the motion information samples.

18. The device of claim 17 , wherein a second amount of OIS is applied to the first image, and wherein the second amount of OIS applied to the first image is based only on an identified first set of high frequency motion information data from the first subset of the motion information samples

Assignees

• Apple Inc

Inventors

• Thivent Damien J
• Williams George E
• Zhou Jianping
• Baer Richard L
• Toft Rolf
• Beysserie Sebastien X

Classifications

• H04N23/6811

  based on the image signal · CPC title

• H04N23/687 Primary

  by shifting the lens or sensor position · CPC title

• H04N23/55 Primary

  Optical parts specially adapted for electronic image sensors; Mounting thereof · CPC title

• H04N23/60

  Control of cameras or camera modules · CPC title

• H04N23/683

  performed by a processor, e.g. controlling the readout of an image memory · CPC title