Automated seamless video loop

Patent metadata

Field	Value
Publication number	US-10062409-B2
Application number	US-201615275105-A
Country	US
Kind code	B2
Filing date	Sep 23, 2016
Priority date	Sep 23, 2016
Publication date	Aug 28, 2018
Grant date	Aug 28, 2018

Abstract

Techniques and devices for creating an AutoLoop output video include identifying optimal loops within short videos or within a series of image. The AutoLoop output video may be automatically created using casually shot, handheld videos, and may include an AutoLoop pipeline that may comprise obtaining an input video, stabilizing the input video, detecting optimal loop parameters and baking out the AutoLoop output video with crossfade and playing back the AutoLoop output video. Video stabilization can include a cascade of video stabilization algorithms including a tripod-direct mode and a tripod-sequential mode. After stabilization, an AutoLoop operation may determine optimal loop parameters. Once optimal loop parameters are determined, a crossfade may be added to smooth out any temporal and spatial discontinuities in the AutoLoop output video.

First claim

The invention claimed is:

1. A non-transitory computer-readable storage media, on which are stored instructions that when executed cause a programmable control device to:
obtain a plurality of frames captured at different points in time;
determine a temporal sequence for the frames based on the different points in time;
apply a stabilization operation to the frames;
determine a set of start frames and a set of corresponding temporal loop periods from the stabilized frames;
select a start frame from the set of start frames and a temporal loop period from the set of corresponding temporal loop periods based on a temporal discontinuity minimization that minimizes frame differences between one or more initial frames of the corresponding temporal loop period and one or more non-looping frames of the temporal sequence that temporally follow one or more last frames of the corresponding temporal loop period;
apply a crossfade to a subset of the stabilized frames that corresponds to the selected start frame and the selected temporal loop period; and
generate an output video loop based on the selected start frame, the selected temporal loop period, and the crossfade.

2. The non-transitory computer-readable storage media of claim 1 , wherein the instructions to apply a stabilization operation comprise instructions that cause the programmable control device to:
select a reference frame from the frames; and
warp coordinates of remaining frames from the frames onto the reference frame to align the remaining frames with the reference frame.

3. The non-transitory computer-readable storage media of claim 2 , wherein the instructions that cause the programmable control device to apply a stabilization operation comprise instructions that cause the programmable control device to map a planar motion of one or more adjacent, pair frames in the frames.

4. The non-transitory computer-readable storage media of claim 1 , wherein the instructions to apply a crossfade comprise instructions that cause the programmable control device to:
obtain one or more additional non-looping frames that temporally occur after the one or more last frames of the selected temporal loop period according to the temporal sequence; and
blend the subset of the stabilized frames with the one or more additional non-looping frames, wherein the subset of the stabilized frames includes the selected start frame.

5. The non-transitory computer-readable storage media of claim 1 , wherein the instructions to apply a stabilization operation comprise instructions that cause the programmable control device to apply a tripod-mode stabilization operation based on a comparison of content between consecutive frames in the frames.

6. The non-transitory computer-readable storage media of claim 1 , wherein the instructions to apply a stabilization operation comprise instructions that cause the programmable control device to apply a tripod-mode stabilization operation based on a comparison of the frames to a reference frame in the frames.

7. The non-transitory computer-readable storage media of claim 1 , wherein the instructions to apply a crossfade comprise instructions that cause the programmable control device to determine a crossfade length based on a second temporal discontinuity minimization that minimizes differences between the crossfade and the subset of the stabilized frames.

8. The non-transitory computer-readable storage media of claim 1 , wherein the instructions to apply a crossfade comprise instructions that cause the programmable control device to apply the crossfade based on a temporal energy function that penalizes differences between the crossfade and the subset of the stabilized frames.

9. The non-transitory computer-readable storage media of claim 1 , wherein the temporal discontinuity minimization is further based on a minimization of a second set of frame differences between a second set of non-looping frames that temporally precedes the one or more initial frames according to the temporal sequence and the one or more last frames within the corresponding temporal loop periods.

10. The non-transitory computer-readable storage media of claim 1 , wherein each pixel within the one or more initial frames and the one or more non-looping frames include a plurality of color channels that are weighted differently, and wherein minimizing the frame differences between the one or more initial frames and the one or more non-looping frames are based on differences between the weighted pixels of the one or more initial frames and the one or more non-looping frames.

11. A computer-implemented method for generating a video loop of frames, comprising:
obtaining an input video;
applying a stabilization operation to the input video;
determining a plurality of start frames and a plurality of corresponding temporal loop periods from the stabilized input video;
selecting a start frame from the plurality of start frames and a temporal loop period from the plurality of corresponding temporal loop periods from the stabilized input video based on a temporal discontinuity minimization that penalizes frame differences between one or more initial frames of the corresponding temporal loop period and one or more non-looping frames of a temporal sequence of the input video that temporally follow one or more last frames of the corresponding temporal loop period;
applying a crossfade to a portion of the stabilized input video that includes the selected start frame and one or more loop frames within the selected temporal loop period, wherein the crossfade includes a subset of frames that temporally occur after the selected temporal loop period according to the temporal sequence; and
generating an output video loop with the selected start frame, the selected temporal loop period, and the crossfade.

12. The computer-implemented method of claim 11 , wherein applying a crossfade comprises:
obtaining the subset of frames after the selected temporal loop period based on a determined length of the crossfade, wherein the determined length is based on a second temporal discontinuity minimization that penalizes differences between the subset of frames and the selected start frame; and
blending the selected start frame with one or more frames from the subset of frames.

13. The computer-implemented method of claim 11 , wherein applying a crossfade comprises applying the crossfade based on a temporal energy function that penalizes differences between the crossfade and the selected start frame.

14. The computer-implemented method of claim 11 , wherein the temporal discontinuity minimization is further based on penalizing a second set of frame differences between a second set of non-looping frames that temporally precedes the one or more initial frames according to the temporal sequence and the one or more last frames of the plurality of corresponding temporal loop periods.

15. The computer-implemented method of claim 11 , wherein each pixel within the one or more initial frames and the one or more non-looping frames include a plurality of color channels that are weighted differently, and wherein the frame differences between the one or more initial frames and the one or more non-looping frames are based on differences between the weighted pixels of the one or more initial frames and the one or more non-looping frames.

16. An electronic device comprising:
an image sensor,
a memory operatively coupled to the image sensor, and
one or more processors operatively coupled to the memory and the image sensor, wherein the one or more processors are programmed to:
obtain a plurality of

Assignees

• Apple Inc

Inventors

• Bradley Arwen V
• Noble Samuel G
• Van Der Merwe Rudolph
• Klivington Jason
• Mitchell Douglas P

Classifications

• G11B27/34

  Indicating arrangements  {(indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D)} · CPC title

• G11B27/007 Primary

  reproducing continuously a part of the information, i.e. repeating · CPC title

• G11B27/031

  Electronic editing of digitised analogue information signals, e.g. audio or video signals · CPC title

• G11B27/19

  by using information detectable on the record carrier · CPC title

• G11B27/034

  on discs (G11B27/036, G11B27/038 take precedence) · CPC title