Systems and methods for generating stabilized visual content using spherical visual content

Patent metadata

Field	Value
Publication number	US-9922398-B1
Application number	US-201615199865-A
Country	US
Kind code	B1
Filing date	Jun 30, 2016
Priority date	Jun 30, 2016
Publication date	Mar 20, 2018
Grant date	Mar 20, 2018

Abstract

Spherical visual content represented in an image space may be obtained. The spherical visual content may have been captured by image sensor(s) during a time duration. The spherical visual content may include phenomena caused by motion of the image sensor(s) and/or optical components that guide light onto the image sensor(s). A capture path taken by the image sensor(s) during the time duration may be determined. The capture path may reflect positions and orientations of the image sensor(s) during the time duration. A smoothed path may be determined based on the capture path. The smoothed path may have smoother changes in positions and/or orientations than the capture path. The image space may be warped based on a difference between the capture path and the smoothed path. The stabilized visual content may be determined by projecting the spherical visual content represented in the warped image space to a spherical projection space.

First claim

What is claimed is:

1. A system for generating stabilized visual content using spherical visual content, the system comprising:
one or more physical processors configured by machine readable instructions to:
obtain the spherical visual content, the spherical visual content captured by one or more image sensors during a time duration, the spherical visual content including phenomena caused by motion of the one or more image sensors and/or one or more optical components that guide light onto the one or more image sensors during at least a part of the time duration, the spherical visual content including pixels represented in an image space, the image space including a projection point inside the image space, wherein the spherical visual content is transformed into a spherical projection space by projecting the pixels in the image space to the spherical projection space along lines including the projection point;
determine a capture path taken by the one or more image sensors during the time duration, the capture path reflecting positions and orientations of the one or more image sensors during the time duration, the capture path including capture viewpoints from which the one or more image sensors captured the spherical visual content during the time duration, the capture path including a first capture viewpoint from which the spherical visual content was captured at a first point in time within the time duration;
determine a smoothed path based on the capture path, the smoothed path having smoother changes in positions and/or orientations than the capture path, the smoothed path including smoothed viewpoints, the smoothed path including a first smoothed viewpoint at the first point in time within the time duration;
warp the image space based on a difference between the capture path and the smoothed path, the difference between the capture path and the smoothed path including a difference between the positions of the first capture viewpoint and the first smoothed viewpoint at the first point in time;
determine the stabilized visual content by projecting the spherical visual content represented in the warped image space to the spherical projection space, wherein views of the stabilized visual content appear to be from the smoothed viewpoints such that a view of the stabilized visual content corresponding to the first point in time appears to be from the first smoothed viewpoint; and
effectuate presentation of the stabilized visual content on a display.

2. The system of claim 1 , wherein the projection point coincides with a center of the spherical visual content represented in the image space.

3. The system of claim 1 , wherein the smoothed path having smoother changes in the positions and/or the orientations than the capture path is characterized by a maximum rate of changes in the positions and/or the orientations of the smoothed path being smaller than a maximum rate of changes in the positions and/or the orientations of the capture path.

4. The system of claim 1 , wherein the smoothed path having smoother changes in the positions and/or the orientations than the capture path is characterized by the smoothed path having less jitters in the positions and/or the orientations than the capture path.

5. The system of claim 1 , wherein the presentation of the stabilized visual content on the display includes playback of the stabilized visual content at a play rate faster than a capture rate at which the one or more image sensors captured the spherical visual content for at least a portion of the time duration.

6. The system of claim 1 , wherein the smoothed viewpoints include a viewing field of view and a viewing rotation of the stabilized visual content, and the presentation of the stabilized visual content on the display includes playback of portions of the stabilized visual content corresponding to the viewing field of view and the viewing rotation of the stabilized visual content.

7. The system of claim 1 , wherein the capture path taken by the one or more image sensors during the time duration is determined based on an analysis of the spherical visual content and/or motion and orientation information for the one or more image sensors, the motion and orientation information for the one or more image sensors generated by a motion and orientation sensor.

8. The system of claim 1 , wherein warping the image space includes warping the image space for a portion of the time duration.

9. The system of claim 1 , wherein the one or more image sensors are carried by an unmanned aerial vehicle.

10. A method for generating stabilized visual content using spherical visual content, the method comprising:
obtaining the spherical visual content, the spherical visual content captured by one or more image sensors during a time duration, the spherical visual content including phenomena caused by motion of the one or more image sensors and/or one or more optical components that guide light onto the one or more image sensors during at least a part of the time duration, the spherical visual content including pixels represented in an image space, the image space including a projection point inside the image space, wherein the spherical visual content is transformed into a spherical projection space by projecting the pixels in the image space to the spherical projection space along lines including the projection point;
determining a capture path taken by the one or more image sensors during the time duration, the capture path reflecting positions and orientations of the one or more image sensors during the time duration, the capture path including capture viewpoints from which the one or more image sensors captured the spherical visual content during the time duration, the capture path including a first capture viewpoint from which the spherical visual content was captured at a first point in time within the time duration;
determining a smoothed path based on the capture path, the smoothed path having smoother changes in positions and/or orientations than the capture path, the smoothed path including smoothed viewpoints, the smoothed path including a first smoothed viewpoint at the first point in time within the time duration;
warping the image space based on a difference between the capture path and the smoothed path, the difference between the capture path and the smoothed path including a difference between the positions of the first capture viewpoint and the first smoothed viewpoint at the first point in time;
determining the stabilized visual content by projecting the spherical visual content represented in the warped image space to the spherical projection space, wherein views of the stabilized visual content appear to be from the smoothed viewpoints such that a view of the stabilized visual content corresponding to the first point in time appears to be from the first smoothed viewpoint; and
effectuating presentation of the stabilized visual content on a display.

11. The method of claim 10 , wherein the projection point coincides with a center of the spherical visual content represented in the image space.

12. The method of claim 10 , wherein the smoothed path having smoother changes in the positions and/or the orientations than the capture path is characterized by a maximum rate of changes in the positions and/or the orientations of the smoothed path being smaller than a maximum rate of changes in the positions and/or the orientations of the capture path.

13. The method of claim 10 , wherein the smoothed path having smoother changes in the positions and/or the orientations than the capture path is characterized by the smoothed path having less jitters in the positions and/or the

Assignees

• Gopro Inc

Inventors

• Jenny Alexandre

Classifications

• G06T7/20

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title

• G06T2207/10032

  Satellite or aerial image; Remote sensing · CPC title

• G06T2207/30244

  Camera pose · CPC title

• G06T2207/30241

  Trajectory · CPC title

• G06T2207/30252

  Vehicle exterior; Vicinity of vehicle · CPC title