Automated seamless video loop

Patent metadata

Field	Value
Publication number	US-2018090171-A1
Application number	US-201615275105-A
Country	US
Kind code	A1
Filing date	Sep 23, 2016
Priority date	Sep 23, 2016
Publication date	Mar 29, 2018
Grant date	—

Abstract

Techniques and devices for creating an AutoLoop output video include identifying optimal loops within short videos or within a series of image. The AutoLoop output video may be automatically created using casually shot, handheld videos, and may include an AutoLoop pipeline that may comprise obtaining an input video, stabilizing the input video, detecting optimal loop parameters and baking out the AutoLoop output video with crossfade and playing back the AutoLoop output video. Video stabilization can include a cascade of video stabilization algorithms including a tripod-direct mode and a tripod-sequential mode. After stabilization, an AutoLoop operation may determine optimal loop parameters. Once optimal loop parameters are determined, a crossfade may be added to smooth out any temporal and spatial discontinuities in the AutoLoop output video.

First claim

1 . A non-transitory computer-readable storage media, on which are stored instructions that when executed cause a programmable control device to:
obtain a plurality of frames captured at different points in time; determine a temporal sequence for the frames based on the different points in time; apply a stabilization operation to the frames; determine a start frame and a temporal loop period from the stabilized frames based on a temporal discontinuity minimization, wherein the temporal discontinuity minimization is based on a minimized frame difference between the start frame and a post-loop frame that temporally follows a last frame within the temporal loop period according to the temporal sequence; apply a crossfade to a subset of the stabilized frames that corresponds to the start frame and the temporal loop period; and generate an output video loop based on the start frame, the temporal loop period, and the crossfade.

2 . The non-transitory computer-readable storage media of claim 1 , wherein the instructions to apply a stabilization operation comprise instructions to:
select a reference frame from the frames; and warp coordinates of remaining frames from the frames onto the reference frame to align the remaining frames with the reference frame.

3 . The non-transitory computer-readable storage media of claim 2 , wherein the instructions that cause the programmable control device to apply a stabilization operation comprise instructions that cause the programmable control device to map a planar motion of one or more adjacent, pair frames in the frames.

4 . The non-transitory computer-readable storage media of claim 1 , wherein the instructions to apply a crossfade comprise instructions that cause the programmable control device to:
obtain one or more additional post-loop frames that temporally occur after the last frame within the temporal loop period according to the temporal sequence; and blend the subset of the stabilized frames with the post-loop frame and the one or more additional post-loop frames, wherein the subset of the stabilized frames includes the start frame.

5 . The non-transitory computer-readable storage media of claim 1 , wherein the instructions to apply a stabilization operation comprise instructions that cause the programmable control device to apply a tripod-mode stabilization operation based on a comparison of content between consecutive frames in the frames.

6 . The non-transitory computer-readable storage media of claim 1 , wherein the instructions to apply a stabilization operation comprise instructions that cause the programmable control device to apply a tripod-mode stabilization operation based on a comparison of the frames to a reference frame in the frames.

7 . The non-transitory computer-readable storage media of claim 1 , wherein the instructions to apply a crossfade comprise instructions that cause the programmable control device to determine a crossfade length based on a second temporal discontinuity minimization that minimizes differences between the crossfade and the subset of the stabilized frames.

8 . The non-transitory computer-readable storage media of claim 1 , wherein the instructions to apply a crossfade comprise instructions that cause the programmable control device to apply the crossfade based on a temporal energy function that penalizes differences between the crossfade and the subset of the stabilized frames.

9 . The non-transitory computer-readable storage media of claim 1 , wherein the temporal discontinuity minimization is further based on a minimized frame difference between a second non-looping frame that temporally precedes the start frame according to the temporal sequence with the last frame within the temporal loop period.

10 . The non-transitory computer-readable storage media of claim 1 , wherein each of pixel within the start frame and the post-loop frame includes a plurality of color channels that are weighted differently, and wherein the minimized frame difference between the start frame and the post-loop frame is based on differences between the weighted pixels of the start frame and the post-loop frame.

11 . A computer-implemented method for generating a video loop of frames, comprising:
obtaining an input video; applying a stabilization operation to the input video; determining a start frame and a temporal loop period from the stabilized input video based on a temporal discontinuity minimization, wherein the temporal discontinuity minimization penalizes a frame difference between the start frame and a post-loop frame that temporally follows a last loop frame within the temporal loop period according to the input video; applying a crossfade to a portion of the stabilized input video that includes the start frame and one or more loop frames within the temporal loop period, wherein the crossfade includes a subset of frames after the temporal loop period according to the input video; and generating an output video loop with the start frame, the temporal loop period, and the crossfade.

12 . The computer-implemented method of claim 11 , wherein applying a crossfade comprises:
obtaining the subset of the frames after the temporal loop period based on a determined length of the crossfade, wherein the determined length is based on a second temporal discontinuity minimization that penalizes differences between the subset of the frames and at least the start frame; and blending the start frame with one or more frames from the subset of the frames.

13 . The computer-implemented method of claim 11 , wherein applying a crossfade comprises applying the crossfade based on a temporal energy function that penalizes differences between the crossfade and at least the start frame.

14 . The computer-implemented method of claim 11 , wherein the temporal discontinuity minimization is further based on penalizing a frame difference between a second non-looping frame that temporally precedes the start frame according to the input video with the last loop frame within the temporal loop period.

15 . The computer-implemented method of claim 11 , wherein each of pixel within the start frame and the post-loop frame includes a plurality of color channels that are weighted differently, and wherein the frame difference between the start frame and the post-loop frame is based on differences between the weighted pixels of the start frame and the post-loop frame.

16 . An electronic device comprising:
an image sensor, a memory operatively coupled to the image sensor, and one or more processors operatively coupled to the memory and the image sensor, wherein the one or more processors are programmed to:
obtain a plurality of frames captured at different points in time;
determine a temporal sequence for the frames based on the different points in time;
determine a start frame and a temporal loop period from the frames based on temporal discontinuity minimization, wherein the temporal discontinuity minimization is based on a frame difference between the start frame and a post-loop frame that temporally follows a last frame within the temporal loop period according to the temporal sequence;
apply a crossfade for a subset of the frames that corresponds to the start frame and the temporal loop period; and
generate an output video loop with the start frame, the temporal loop period, and the crossfade.

17 . The electronic device of claim 16 , wherein the one or more processors being programmed to apply a crossfade causes the one or more processors to be programmed to:
obtain one or more additional post-loop frame

Assignees

• Apple Inc

Inventors

• Bradley Arwen V
• Noble Samuel G
• Van Der Merwe Rudolph
• Klivington Jason
• Mitchell Douglas P

Classifications

• G11B27/031

  Electronic editing of digitised analogue information signals, e.g. audio or video signals · CPC title

• G11B27/19

  by using information detectable on the record carrier · CPC title

• G11B27/34

  Indicating arrangements  {(indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D)} · CPC title

• G11B27/007 Primary

  reproducing continuously a part of the information, i.e. repeating · CPC title

• G11B27/034

  on discs (G11B27/036, G11B27/038 take precedence) · CPC title