Adjusting field of view of truncated square pyramid projection for 360-degree video

Patent metadata

Field	Value
Publication number	US-2018199029-A1
Application number	US-201815862468-A
Country	US
Kind code	A1
Filing date	Jan 4, 2018
Priority date	Jan 11, 2017
Publication date	Jul 12, 2018
Grant date	—

Abstract

Techniques and systems are described for encoding 360-degree video data using the planes of a truncated square pyramid to map the 360-degree data for different fields of view. 360-degree video data can include multiple frames, where each frame includes spherical video data. In various implementations, a video coding system can select a field of view for the video data, and determine an offset from the center of the spherical video data that corresponds to the field of view. Using the offset, the system can determine a projection of the spherical video data onto the planes of the truncated square pyramid, where the base plane represents a front view and the top plane represents a back view. The system can then map the video data according to the projection such that each plane of the truncated square pyramid includes a portion of the spherical video data.

First claim

What is claimed is:

1 . A method for encoding video data, comprising:
obtaining virtual reality video data representing a 360-degree view of a virtual environment, wherein the virtual reality video data includes a plurality of frames, and wherein each frame from the plurality of frames includes corresponding spherical video data; determining a field of view for a frame from the plurality of frames; determining an offset from a center of the spherical video data for the frame, the offset corresponding to the field of view; determining, using the offset, a projection of the spherical video data onto planes of a truncated square pyramid, wherein a base plane of the truncated square pyramid represents a front view of the spherical video data, the front view corresponding to the field of view, wherein a top plane of the truncated square pyramid represents a back view of the spherical video data, and wherein the top plane is smaller than the base plane; and mapping, according to the projection, the spherical video data to the planes of the truncated square pyramid, wherein each plane includes a portion of the spherical video data.

2 . The method of claim 1 , wherein determining the projection includes:
determining a point on a plane from the planes of the truncated square pyramid; and determining a pixel from the spherical video data that corresponds to the point on the plane.

3 . The method of claim 1 , wherein the planes of the truncated square pyramid include a first side plane, a second side plane, a third side plane, and a fourth side plane, wherein each of the first side plane, the second side plane, the third side plane, and the fourth side plane include one edge that is adjacent to the base plane, one edge that is adjacent to the top plane, and two edges that are adjacent to two other of the first side plane, the second side plane, the third side plane, and the fourth side plane.

4 . The method of claim 3 , wherein a resolution of each of the first side plane, the second side plane, the third side plane, and the fourth side plane decreases from the edge adjacent to the base plane to the edge adjacent to the top plane.

5 . The method of claim 1 , wherein the top plane has a lower resolution than the base plane.

6 . The method of claim 1 , wherein the top plane has a minimum size that is irrespective of the offset, wherein the minimum size is greater than zero.

7 . The method of claim 1 , wherein the offset corresponds to a field of view that makes a size of the top plane zero.

8 . The method of claim 1 , wherein determining the projection includes:
determining, using the offset, a first projection of a first portion of data from the spherical video data onto the base plane; and determining, using the offset, a second projection of a second portion of data from the spherical video data onto the top plane.

9 . The method of claim 1 , wherein determining the projection includes:
determining, using a size of the base plane and a size of the top plane, a third projection of a third portion of data from the spherical video data onto a first side plane, the third projection including a view above the field of view; determining, using a size of the base plane and a size of the top plane, a fourth projection of a fourth portion of data from the spherical video data onto a second side plane, the fourth projection including a view below the field of view; determining, using a size of the base plane and a size of the top plane, a fifth projection of a fifth portion of data from the spherical video data onto a third side plane, the fifth projection including a view to the right of the field of view; and determining, using a size of the base plane and a size of the top plane, a sixth projection of a sixth portion of data from the spherical video data onto a fourth side plane, the sixth projection including a view to the left of the field of view.

10 . The method of claim 1 , wherein the field of view is greater than 90 degrees.

11 . The method of claim 1 , further comprising:
packing each of the planes in a rectangular format.

12 . A device for coding video data, comprising:
a memory configured to store video data; and a video coding device in communication with the memory, wherein the video coding device is configured to:
obtain virtual reality video data representing a 360-degree view of a virtual environment, wherein the virtual reality video data includes a plurality of frames, and wherein each frame from the plurality of frames includes corresponding spherical video data;
determine a field of view for a frame from the plurality of frames;
determine an offset from a center of the spherical video data for the frame, the offset corresponding to the field of view;
determine, using the offset, a projection of the spherical video data onto planes of a truncated square pyramid, wherein a base plane of the truncated square pyramid represents a front view of the spherical video data, the front view corresponding to the field of view, wherein a top plane of the truncated square pyramid represents a back view of the spherical video data, and wherein the top plane is smaller than the base plane; and
map, according to the projection, the spherical video data to the planes of the truncated square pyramid, wherein each plane includes a portion of the spherical video data.

13 . The device of claim 12 , wherein determining the projection includes:
determining a point on a plane from the planes of the truncated square pyramid; and determining a pixel from the spherical video data that corresponds to the point on the plane.

14 . The device of claim 12 , wherein the planes of the truncated square pyramid include a first side plane, a second side plane, a third side plane, and a fourth side plane, wherein each of the first side plane, the second side plane, the third side plane, and the fourth side plane include one edge that is adjacent to the base plane, one edge that is adjacent to the top plane, and two edges that are adjacent to two other of the first side plane, the second side plane, the third side plane, and the fourth side plane.

15 . The device of claim 14 , wherein a resolution of each of the first side plane, the second side plane, the third side plane, and the fourth side plane decreases from an edge adjacent to the base plane to an edge adjacent to the top plane.

16 . The device of claim 12 , wherein the top plane has a lower resolution than the base plane.

17 . The device of claim 12 , wherein the top plane has a minimum size that is irrespective of the offset, wherein the minimum size is greater than zero.

18 . The device of claim 12 , wherein the offset corresponds to a field of view that makes a size of the top plane zero.

19 . The device of claim 12 , wherein determining the projection includes:
determining, using the offset, a first projection of a first portion of data from the spherical video data onto the base plane; and determining, using the offset, a second projection of a second portion of data from the spherical video data onto the top plane.

20 . The device of claim 12 , wherein determining the projection includes:
determining, using a size of the base plane and a size of the top plane, a third projection of a third portion of data from the spherical video data onto a first side plane, the third projection including a view above the field of view; determining, using a size of the base plane and a size of the top plane, a fourth projection of a

Assignees

• Qualcomm Inc

Inventors

• Van Der Auwera Geert
• Coban Muhammed

Classifications

• H04N13/363 Primary

  using image projection screens (volumetric display H04N13/388) · CPC title

• H04N13/161

  Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597) · CPC title

• H04N13/111

  Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation · CPC title

• G06T3/00 Primary

  Geometric image transformations in the plane of the image · CPC title

• H04N5/2628

  Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation · CPC title