Spherical video stabilization based on accelerometer data

Patent metadata

Field	Value
Publication number	US-10084962-B2
Application number	US-201615353488-A
Country	US
Kind code	B2
Filing date	Nov 16, 2016
Priority date	Nov 16, 2015
Publication date	Sep 25, 2018
Grant date	Sep 25, 2018

Abstract

A method includes identifying a frame of a spherical video as a key frame, storing a compensation component based on position data of a camera in association with the key frame, and compensating for a movement of the camera in a subsequent frame of the spherical video based on the key frame and the compensation component associated with the key frame.

First claim

What is claimed is:

1. A method comprising:
identifying a frame of a spherical video as a key frame;
storing a compensation component based on position data of a camera in association with the key frame;
compensating for a movement of the camera in a subsequent frame of the spherical video based on the key frame and the compensation component associated with the key frame; and
in response to determining the spherical video was captured as partial spherical data, set a pixel of the movement compensated subsequent frame outside a display region to a default value.

2. The method of claim 1 , further comprising:
capturing accelerometer data for the camera with each captured frame of the spherical video;
using the accelerometer data to generate at least one of a center position, a translation position and a rotation position associated with the key frame; and
storing the at least one of the center position, the translation position and the rotation position associated with the key frame, wherein the compensation component is based on the at least one of the center position, the translation position and the rotation position.

3. The method of claim 1 , further comprising:
matching a portion of the subsequent frame to a plurality of portions of the key frame;
selecting one of the plurality of portions of the key frame for use in the compensating for the movement of the camera based on a direction of the compensation component; and
compensating for the movement of the camera in the subsequent frame using the selected one of the plurality of portions of the key frame.

4. The method of claim 1 , wherein
the key frame is one of a plurality of key frames, and
the compensation component associated with the key frame is further based on a previous key frame of the plurality of key frames.

5. The method of claim 1 , wherein the compensating for the camera movement is performed by one of a camera, a server configured to stream the spherical video and a device configured to render the spherical video.

6. The method of claim 1 , wherein
the compensation component includes a center position and a translation position associated with the key frame, and
the compensating for the camera movement includes centering the subsequent frame based on the key frame center position and the key frame translation position.

7. The method of claim 1 , wherein
the subsequent frame of the spherical video is stored as three dimensional (3D) data,
the compensation component includes a center position and a translation position associated with the key frame, and
the compensating for the camera movement includes rotating the subsequent frame in 3D based on the key frame center position and the key frame translation position.

8. The method of claim 1 , wherein
the subsequent frame of the spherical video is stored as two dimensional (2D) data,
the compensation component includes a center position and a translation position associated with the key frame, and
the compensating for the camera movement includes shifting the subsequent frame in 2D based on the key frame center position and the key frame translation position, the method further comprising:
correcting for border image loss.

9. The method of claim 1 , wherein
the subsequent frame of the spherical video is stored as two dimensional (2D) data,
the compensation component includes a center position and a translation position associated with the key frame,
the compensating for the camera movement includes shifting the subsequent frame in 2D based on the key frame center position and the key frame translation position;
upon determining the spherical video was not captured as partial spherical data, correcting for border image loss.

10. A device comprising;
a processor;
a memory configured to store instructions that when executed by the processor cause the device to:
identify a frame of a spherical video as a key frame;
store a compensation component based on position data in association with the key frame;
compensate for a camera movement in a subsequent frame of the spherical video based on the key frame and the compensation component associated with the key frame; and
in response to determining the spherical video was captured as partial spherical data, set pixels of the movement compensated subsequent frame outside a display region of the device to a default value.

11. The device of claim 10 , wherein the device is one of a camera, a server configured to stream the spherical video and a device configured to render the spherical video.

12. The device of claim 10 , wherein the instructions further cause the processor to:
capture accelerometer data for the camera with each captured frame of the spherical video; and
use the accelerometer data to generate at least one of a center position, a translation position and a rotation position associated with the key frame; and
store the at least one of the center position, the translation position and the rotation position associated with the key frame, wherein the compensation component is based on the at least one of the center position, the translation position and the rotation position.

13. The device of claim 10 , wherein the instructions further cause the processor to:
match a portion of the subsequent frame to a plurality of portions of the key frame;
select one of the plurality of portions of the key frame for use in the compensating for the movement of the camera based on a direction of the compensation component, and
compensate for the movement of the camera in the subsequent frame using the selected one of the plurality of portions of the key frame.

14. The device of claim 10 , wherein
the key frame is one of a plurality of key frames, and
the compensation component associated with the key frame is further based on a previous key frame of the plurality of key frames.

15. The device of claim 10 , wherein
the compensation component includes a center position and a translation position associated with the key frame, and
the compensating for the camera movement includes centering the subsequent frame based on the key frame center position and the key frame translation position.

16. The device of claim 10 , wherein
subsequent frame of the spherical video is stored as three dimensional (3D) data,
the compensation component includes a center position and a translation position associated with the key frame, and
the compensating for the camera movement includes rotating the subsequent frame in 3D based on the key frame center position and the key frame translation position.

17. The device of claim 10 , wherein
subsequent frame of the spherical video is stored as two dimensional (2D) data,
the compensation component includes a center position and a translation position associated with the key frame, and
the compensating for the camera movement includes shifting the subsequent frame in 2D based on the key frame center position and the key frame translation position, the method further comprising:
correcting for border image loss.

18. The device of claim 10 , wherein
the subsequent frame of the spherical video is stored as two dimensional (2D) data,
the compensation component includes a center position and a translation position associated with the key frame,
the compensating for the camera movement includes shifting the subsequent frame in 2D based on the key frame center position and the key frame translation position, and
upon determining the spherical video was not captured as partial spherical data, correcting fo

Assignees

• Google Inc
• Google Llc

Inventors

• Kokaram Anil Christopher
• Bengali Husain
• Birkbeck Neil Aylon Charles
• Inguva Sasi

Classifications

• H04N23/698

  for achieving an enlarged field of view, e.g. panoramic image capture · CPC title

• H04N23/6812

  based on additional sensors, e.g. acceleration sensors · CPC title

• H04N23/6845 Primary

  by combination of a plurality of images sequentially taken · CPC title

• H04N23/683 Primary

  performed by a processor, e.g. controlling the readout of an image memory · CPC title

• H04N23/6815

  by distinguishing pan or tilt from motion · CPC title