Systems and methods for stabilizing videos

Patent metadata

Field	Value
Publication number	US-12256147-B2
Application number	US-202318345229-A
Country	US
Kind code	B2
Filing date	Jun 30, 2023
Priority date	May 18, 2018
Publication date	Mar 18, 2025
Grant date	Mar 18, 2025

Abstract

Images with an optical field of view are captured by an image capture device. An observed trajectory of the image capture device reflects the positions of the image capture device at different moments may be determined. A capture trajectory of the image capture device reflects virtual positions of the image capture device from which video content may be generated. The capture trajectory is determined based on a subsequent portion of the observed trajectory such that a portion of the capture trajectory corresponding to a portion of the observed trajectory is determined based on a subsequent portion of the observed trajectory. Orientations of punch-outs for the images are determined based on the capture trajectory. Video content is generated based on visual content of the images within the punch-outs.

First claim

What is claimed is:

1. A system for stabilizing videos, the system comprising:
one or more physical processors configured by machine-readable instructions to:
obtain images captured by an image capture device during a capture duration, rotational positions of the image capture device during the capture duration reflected within a trajectory of the image capture device;
stabilize the trajectory of the image capture device based on a look ahead of the trajectory of the image capture device and a set of constraints, wherein changes in the rotational positions of the image capture device during the capture duration are smoothed in the stabilized trajectory of the image capture device, wherein the trajectory of the image capture device is stabilized to generate a path that minimizes a combination of rotational velocity and rotational acceleration of the image capture device while respecting the set of constraints, the rotational velocity of the image capture device including a time derivative of the rotational positions of the image capture device and the rotational acceleration of the image capture device including a second time derivative of the rotational positions of the image capture device; and
generate video frames of a stabilized video based on the images and the stabilized trajectory of the image capture device.

2. The system of claim 1 , wherein the stabilization of the trajectory of the image capture device based on the look ahead of the trajectory of the image capture device includes use of a subsequent portion of the trajectory of the image capture device to determine whether motion of the image capture device in a preceding portion of the trajectory of the image capture device is intended or not.

3. The system of claim 2 , wherein intended motion of the image capture device is preserved in the stabilized trajectory of the image capture device.

4. The system of claim 1 , wherein the generation of the video frames of the stabilized video based on the images and the stabilized trajectory of the image capture device includes:
determination of punch-outs of the images based on the stabilized trajectory of the image capture device; and
generation of the video frames of the stabilized video based on the punch-outs of the images.

5. The system of claim 1 , wherein the rotational velocity and the rotational acceleration of the image capture device have different affect on the minimization of the combination of the rotational velocity and the rotational acceleration of the image capture device for the generation of the path.

6. The system of claim 1 , wherein the set of constraints includes a margin constraint, the margin constraint providing a limitation on deviation of the stabilized trajectory of the image capture device from the trajectory of the image capture device.

7. The system of claim 1 , wherein the set of constraints includes a target constraint, the target constraint requiring a target in the images to be within punch-outs of the images.

8. The system of claim 1 , wherein the images are stored in a buffer during the stabilization of the trajectory of the image capture device such that individual ones of the images corresponding to a portion of the trajectory of the image capture device are stored in the buffer during the stabilization of the portion of the trajectory of the image capture device.

9. A method for stabilizing videos, the method performed by a computing system including one or more processors, the method comprising:
obtaining, by the computing system, images captured by an image capture device during a capture duration, rotational positions of the image capture device during the capture duration reflected within a trajectory of the image capture device;
stabilizing, by the computing system, the trajectory of the image capture device based on a look ahead of the trajectory of the image capture device and a set of constraints, wherein changes in the rotational positions of the image capture device during the capture duration are smoothed in the stabilized trajectory of the image capture device, wherein the trajectory of the image capture device is stabilized to generate a path that minimizes a combination of rotational velocity and rotational acceleration of the image capture device while respecting the set of constraints, the rotational velocity of the image capture device including a time derivative of the rotational positions of the image capture device and the rotational acceleration of the image capture device including a second time derivative of the rotational positions of the image capture device; and
generating, by the computing system, video frames of a stabilized video based on the images and the stabilized trajectory of the image capture device.

10. The method of claim 9 , wherein stabilizing the trajectory of the image capture device based on the look ahead of the trajectory of the image capture device includes using a subsequent portion of the trajectory of the image capture device to determine whether motion of the image capture device in a preceding portion of the trajectory of the image capture device is intended or not.

11. The method of claim 10 , wherein intended motion of the image capture device is preserved in the stabilized trajectory of the image capture device.

12. The method of claim 9 , wherein generating the video frames of the stabilized video based on the images and the stabilized trajectory of the image capture device includes:
determining punch-outs of the images based on the stabilized trajectory of the image capture device; and
generating the video frames of the stabilized video based on the punch-outs of the images.

13. The method of claim 9 , wherein the rotational velocity and the rotational acceleration of the image capture device have different affect on the minimization of the combination of the rotational velocity and the rotational acceleration of the image capture device for the generation of the path.

14. The method of claim 9 , wherein the set of constraints includes a margin constraint, the margin constraint providing a limitation on deviation of the stabilized trajectory of the image capture device from the trajectory of the image capture device.

15. The method of claim 9 , wherein the set of constraints includes a target constraint, the target constraint requiring a target in the images to be within punch-outs of the images.

16. The method of claim 9 , wherein the images are stored in a buffer during the stabilization of the trajectory of the image capture device such that individual ones of the images corresponding to a portion of the trajectory of the image capture device are stored in the buffer during the stabilization of the portion of the trajectory of the image capture device.

17. A system for stabilizing videos, the system comprising:
one or more physical processors configured by machine-readable instructions to:
obtain images captured by an image capture device during a capture duration, rotational positions of the image capture device during the capture duration reflected within a trajectory of the image capture device, the trajectory of the image capture device including a first portion corresponding to a first moment within the capture duration and a second portion corresponding to a second moment subsequent to the first moment within the capture duration;
stabilize the trajectory of the image capture device based on a set of constraints and a look ahead of the trajectory of the image capture device by using a subsequent portion of the trajectory of the image capture device to stabilize a prece

Assignees

• Gopro Inc

Inventors

• Derbanne Thomas
• Douady Cesar
• Karpushin Maxim

Classifications

• H04N23/51

  Housings · CPC title

• H04N23/683 Primary

  performed by a processor, e.g. controlling the readout of an image memory · CPC title

• H04N23/6812 Primary

  based on additional sensors, e.g. acceleration sensors · CPC title