Systems and methods for combining multiple frames to produce media content with simulated exposure effects

Patent metadata

Field	Value
Publication number	US-9774784-B2
Application number	US-201615295127-A
Country	US
Kind code	B2
Filing date	Oct 17, 2016
Priority date	Dec 17, 2014
Publication date	Sep 26, 2017
Grant date	Sep 26, 2017

Abstract

Systems, methods, and non-transitory computer-readable media can capture media content including an original set of frames. A plurality of subsets of frames can be identified, based on a subset selection input, out of the original set of frames. An orientation-based image stabilization process can be applied to each subset in the plurality of subsets of frames to produce a plurality of stabilized subsets of frames. Multiple frames within each stabilized subset in the plurality of stabilized subsets of frames can be combined to produce a plurality of combined frames. Each stabilized subset of frames can be utilized to produce a respective combined frame in the plurality of combined frames. A time-lapse media content item can be provided based on the plurality of combined frames.

First claim

What is claimed is:

1. A computer-implemented method comprising:
determining, by a computing system, a plurality of subsets of frames of an original set of frames;
applying, by the computing system, an orientation-based image stabilization process to each subset in the plurality of subsets of frames to produce a plurality of stabilized subsets of frames, wherein the applying the orientation-based image stabilization process comprises minimizing a rate of rotation between successive frames within each subset while minimizing empty regions below a threshold;
combining, by the computing system, multiple frames within each stabilized subset in the plurality of stabilized subsets of frames to produce a plurality of combined frames, wherein each stabilized subset of frames is utilized to produce a respective combined frame in the plurality of combined frames; and
generating, by the computing system, a time-lapse media content item based on the plurality of combined frames.

2. The computer-implemented method of claim 1 , wherein the applying of the orientation-based image stabilization process to each subset in the plurality of subsets of frames to produce the plurality of stabilized subsets of frames further comprises:
acquiring timestamps for multiple frames within each subset in the plurality of subsets of frames;
acquiring camera orientation data associated with a camera utilized to capture a media content including the original set of frames and associated with the timestamps for the multiple frames within each subset;
generating a smoothed set of camera orientation data by minimizing the rate of rotation between the successive frames within each subset while minimizing the empty regions below the threshold; and
warping the multiple frames within each subset based on the smoothed set of camera orientation data to produce each stabilized subset in the plurality of stabilized subsets of frames.

3. The computer-implemented method of claim 2 , wherein the orientation-based image stabilization process causes the multiple frames within each subset to be aligned relative to at least one of a respective first frame of the multiple frames within each subset or a specified alignment.

4. The computer-implemented method of claim 2 , wherein the camera incurs undesirable motion while capturing the media content including the original set of frames, and wherein the orientation-based image stabilization process reduces instability, caused by the undesirable motion, with respect to each subset in the plurality of subsets of frames.

5. The computer-implemented method of claim 1 , wherein a moving object is represented in at least one stabilized subset in the plurality of stabilized subsets, wherein the combining of the multiple frames within each stabilized subset comprises combining multiple frames within the at least one stabilized subset, and wherein the combining of the multiple frames within the at least one stabilized subset causes one or more motion trails associated with the moving object to appear in a combined frame produced based on the at least one stabilized sub set.

6. The computer-implemented method of claim 1 , wherein the combining of the multiple frames within each stabilized subset in the plurality of stabilized subsets of frames to produce the plurality of combined frames further comprises:
blending pixels included in the multiple frames within each stabilized subset.

7. The computer-implemented method of claim 6 , wherein the blending of the pixels included in the multiple frames within each stabilized subset further comprises:
calculating at least one of average pixel values for the pixels included in the multiple frames within each stabilized subset, maximum pixel values for the pixels included in the multiple frames within each stabilized subset, or minimum pixel values for the pixels included in the multiple frames within each stabilized subset.

8. The computer-implemented method of claim 1 , further comprising obtaining a subset selection input, wherein the subset selection input corresponds to at least one of a user command or a system setting, and wherein the subset selection input is obtained prior to at least one of the determining of the plurality of subsets of frames or the generating of the time-lapse media content item.

9. The computer-implemented method of claim 8 , wherein the subset selection input indicates a quantity of frames, and wherein the subset selection input causes each subset in the plurality of subsets to include the quantity of frames.

10. The computer-implemented method of claim 9 , wherein the quantity of frames is greater than one.

11. A system comprising:
at least one processor; and
a memory storing instructions that, when executed by the at least one processor, cause the system to perform:
determining a plurality of subsets of frames of an original set of frames;
applying an orientation-based image stabilization process to each subset in the plurality of subsets of frames to produce a plurality of stabilized subsets of frames, wherein the applying the orientation-based image stabilization process comprises minimizing a rate of rotation between successive frames within each subset while minimizing empty regions below a threshold;
combining multiple frames within each stabilized subset in the plurality of stabilized subsets of frames to produce a plurality of combined frames, wherein each stabilized subset of frames is utilized to produce a respective combined frame in the plurality of combined frames; and
generating a time-lapse media content item based on the plurality of combined frames.

12. The system of claim 11 , wherein the applying of the orientation-based image stabilization process to each subset in the plurality of subsets of frames to produce the plurality of stabilized subsets of frames further comprises:
acquiring timestamps for multiple frames within each subset in the plurality of subsets of frames;
acquiring camera orientation data associated with a camera utilized to capture a media content including the original set of frames and associated with the timestamps for the multiple frames within each subset;
generating a smoothed set of camera orientation data by minimizing the rate of rotation between the successive frames within each subset while minimizing the empty regions below the threshold; and
warping the multiple frames within each subset based on the smoothed set of camera orientation data to produce each stabilized subset in the plurality of stabilized subsets of frames.

13. The system of claim 11 , wherein a moving object is represented in at least one stabilized subset in the plurality of stabilized subsets, wherein the combining of the multiple frames within each stabilized subset comprises combining multiple frames within the at least one stabilized subset, and wherein the combining of the multiple frames within the at least one stabilized subset causes one or more motion trails associated with the moving object to appear in a combined frame produced based on the at least one stabilized subset.

14. The system of claim 11 , wherein the combining of the multiple frames within each stabilized subset in the plurality of stabilized subsets of frames to produce the plurality of combined frames further comprises:
blending pixels included in the multiple frames within each stabilized subset.

15. The system of claim 14 , wherein the blending of the pixels included in the multiple frames within each stabilized subset further comprises:
calculating at least one of average pixel values for the pixels included in the multiple frames within each sta

Assignees

• Facebook Inc

Inventors

• Karpenko Alexandre
• Dimson Thomas

Classifications

• G06T5/50 Primary

  using two or more images, e.g. averaging or subtraction · CPC title

• H04N23/683

  performed by a processor, e.g. controlling the readout of an image memory · CPC title

• H04N23/68

  for stable pick-up of the scene, e.g. compensating for camera body vibrations · CPC title

• H04N23/6812

  based on additional sensors, e.g. acceleration sensors · CPC title

• H04N23/6845

  by combination of a plurality of images sequentially taken · CPC title