Time-lapse video capture with optimal image stabilization

Patent metadata

Field	Value
Publication number	US-9426409-B2
Application number	US-201514613133-A
Country	US
Kind code	B2
Filing date	Feb 3, 2015
Priority date	Sep 30, 2014
Publication date	Aug 23, 2016
Grant date	Aug 23, 2016

Abstract

Traditionally, time-lapse videos are constructed from images captured at time intervals called “temporal points of interests” or “temporal POIs.” Disclosed herein are systems and methods of constructing improved, motion-stabilized time-lapse videos using temporal points of interest and image similarity comparisons. According to some embodiments, a “burst” of images may be captured, centered around the aforementioned temporal points of interest. Then, each burst sequence of images may be analyzed, e.g., by performing an image similarity comparison between each image in the burst sequence and the image selected at the previous temporal point of interest. Selecting the image from a given burst that is most similar to the previous selected image, while minimizing the amount of motion with the previous selected image, allows the system to improve the quality of the resultant time-lapse video by discarding “outlier” or other undesirable images captured in the burst sequence and motion stabilizing the selected image.

First claim

The invention claimed is:

1. A non-transitory program storage device, readable by a programmable control device and comprising instructions stored thereon to cause one or more processing units to:
acquire a first digital image from an image sensor at a first time;
acquire a plurality of second digital images from the image sensor, wherein acquisition times of the plurality of second digital images are temporally centered around a determined second time;
determine a similarity value for each of the plurality of second digital images, wherein the similarity values comprise an assessment of the similarity of a respective image from the plurality of second digital images and the first digital image;
discard, from the plurality of second digital images, those images having a similarity value below a threshold value;
select, from among the non-discarded images of the plurality of second digital images, a second digital image that minimizes a motion difference with the first digital image; and
combine the first digital image and the selected second digital image into a time lapse video.

2. The non-transitory program storage device of claim 1 , wherein the similarity values are determined based, at least in part, on an evaluation of at least one of the following characteristics of a second digital image: a color histogram, an exposure level, a focus level, a feature detection process, an image registration process, a pixel sum process, a ΔE process, an auto focus oscillation level, and an average luminance level.

3. The non-transitory program storage device of claim 1 , wherein the instructions to discard, from the plurality of second digital images, those images having a similarity value below a threshold value further comprise instructions to: delete each discarded digital image from the plurality of second digital images before causing the one or more processing units to combine the first digital image and the selected second digital image into a time lapse video.

4. The non-transitory program storage device of claim 1 , further comprising instructions stored thereon to cause the one or more processing units to:
apply image stabilization to the selected second digital image.

5. The non-transitory program storage device of claim 4 , wherein a strength of the image stabilization applied to the selected second digital image is based, at least in part, on an estimated amount of motion blur in the selected second digital image.

6. The non-transitory program storage device of claim 1 , further comprising instructions stored thereon to cause the one or more processing units to:
weight at least one of the determined similarity values for a respective second digital image based, at least in part, on a content of the respective second digital image.

7. The non-transitory program storage device of claim 1 , further comprising instructions stored thereon to cause the one or more processing units to:
weight at least one of the determined similarity values for a respective second digital image based, at least in part, on a time of capture of the respective second digital image with respect to the determined second time.

8. A computer-implemented method, comprising:
acquiring a first digital image from an image sensor at a first time;
acquiring a plurality of second digital images from the image sensor, wherein acquisition times of the plurality of second digital images are temporally centered around a determined second time;
determining a similarity value for each of the plurality of second digital images, wherein the similarity values comprise an assessment of the similarity of a respective image from the plurality of second digital images and the first digital image;
discarding, from the plurality of second digital images, those images having a similarity value below a threshold value;
selecting, from among the non-discarded images of the plurality of second digital images, a second digital image that minimizes a motion difference with the first digital image; and
combining the first digital image and the selected second digital image into a time lapse video.

9. The method of claim 8 , wherein the similarity values are determined based, at least in part, on an evaluation of at least one of the following characteristics of a second digital image: a color histogram, an exposure level, a focus level, a feature detection process, an image registration process, a pixel sum process, a ΔE process, an auto focus oscillation level, and an average luminance level.

10. The method of claim 8 , wherein the act of discarding, from the plurality of second digital images, those images having a similarity value below a threshold value further comprises: deleting each discarded digital image from the plurality of second digital images.

11. The method of claim 8 , further comprising applying image stabilization to the selected second digital image, wherein a strength of the image stabilization applied to the selected second digital image is based, at least in part, on an estimated amount of motion blur in the selected second digital image.

12. The method of claim 8 , further comprising weighting at least one of the determined similarity values for a respective second digital image based, at least in part, on a content of the respective second digital image.

13. The method of claim 8 , further comprising weighting at least one of the determined similarity values for a respective second digital image based, at least in part, on a time of capture of the respective second digital image with respect to the determined second time.

14. An electronic device comprising:
an image sensor,
a memory operatively coupled to the image sensor, and
at least one processor operatively coupled to the memory and the image sensor, wherein the at least one processor is programmed to:
cause the image sensor to acquire a first digital image at a first time;
cause the image sensor to acquire a plurality of second digital images, wherein acquisition times of the plurality of second digital images are temporally centered around a determined second time;
determine a similarity value for each of the plurality of second digital images, wherein the similarity values comprise an assessment of the similarity of a respective image from the plurality of second digital images and the first digital image;
discard, from the plurality of second digital images, those images having a similarity value below a threshold value;
select, from among the non-discarded images of the plurality of second digital images, a second digital image that minimizes a motion difference with the first digital image; and
combine the first digital image and the selected second digital image into a time lapse video.

15. The electronic device of claim 14 , wherein the similarity values are determined based, at least in part, on an evaluation of at least one of the following characteristics of a second digital image: a color histogram, an exposure level, a focus level, a feature detection process, an image registration process, a pixel sum process, a ΔE process, an auto focus oscillation level, and an average luminance level.

16. The electronic device of claim 14 , wherein the instructions to discard, from the plurality of second digital images, those images having a similarity value below a threshold value that the at least one processor is programmed to perform further comprise instructions to: delete each discarded digital image from the plurality of second digital images before the at least one processor combines the first digital image and the selected second digital ima

Assignees

• Apple Inc

Inventors

• Beysserie Sebastien X
• Klivington Jason
• Toft Rolf
• Doepke Frank

Classifications

• G11B15/1875

  adaptations for special effects or editing (signal processing or indexing therefor G11B27/00) · CPC title

• H04N5/147

  Scene change detection · CPC title

• H04N5/915

  for field- or frame-skip recording or reproducing · CPC title

• H04N5/783

  Adaptations for reproducing at a rate different from the recording rate · CPC title

• H04N5/77

  between a recording apparatus and a television camera · CPC title