Method and apparatus for streaming multi-view 360 degree video

Patent metadata

Field	Value
Publication number	US-11575871-B2
Application number	US-202117500439-A
Country	US
Kind code	B2
Filing date	Oct 13, 2021
Priority date	Oct 14, 2020
Publication date	Feb 7, 2023
Grant date	Feb 7, 2023

Abstract

Provided are a method and an apparatus for streaming a multi-view 360 degree video, and a method for streaming a 360 degree video according to an embodiment of the present disclosure includes: encoding a multi-view video to a bitstream of a base layer and a bitstream of a tile layer constituted by at least one tile; selecting a tile included in a user view video in the encoded bitstream of the tile layer by using user view information received from a 360 degree video rendering apparatus, and video information of the multi-view video; extracting tile data included in the selected user view video from the encoded bitstream of the tile layer, and generating a tile bitstream corresponding to the extracted tile data; and transmitting the encoded bitstream of the base layer and the generated tile bitstream to the 360 degree video rendering apparatus.

First claim

What is claimed is:

1. A method for streaming a 360 degree video, which is performed by a 360 degree video streaming apparatus, the method comprising:
encoding a multi-view video to a bitstream of a base layer and a bitstream of a tile layer constituted by at least one tile;
selecting a tile included in a user view video in the encoded bitstream of the tile layer by using user view information received from a 360 degree video rendering apparatus, and video information of the multi-view video;
extracting tile data included in the selected user view video from the encoded bitstream of the tile layer, and generating a tile bitstream corresponding to the extracted tile data; and
transmitting the encoded bitstream of the base layer and the generated tile bitstream to the 360 degree video rendering apparatus,
wherein in the selecting, a pixel warped to the user view video from the multi-view video is calculated by using a rotation matrix representing a rotation to the user view video from the multi-view video and a translation matrix representing a distance.

2. The method of claim 1 , wherein the user view information includes rotation and position values of a head-mounted display worn by a user.

3. The method of claim 1 , wherein the video information of the multi-view video includes a depth map of the multi-view video and a camera parameter of each video.

4. The method of claim 1 , wherein in the selecting, some pixels corresponding to each other are selected in the multi-view video and the user view video, respectively to calculate whether a user view tile is included.

5. The method of claim 4 , wherein a coordinate of a pixel to be used for calculating whether the user view tile is included is determined as a multiple of a Heuristic interval coefficient.

6. The method of claim 1 , wherein in the selecting, a tile index of a pixel coordinate in the multi-view video corresponding to the calculated warped pixel is calculated by using a width and a height of the multi-view video and a width and a height of the tile.

7. The method of claim 1 , wherein in the selecting, when the calculated warped pixel is included in a point set of the user view video, the tile index of the pixel coordinate in the multi-view video corresponding to the calculated warped pixel is added to an index set of the user view tile.

8. The method of claim 7 , wherein in the generating, tile data corresponding to the added index set of the user view tile is extracted from the encoded bitstream of the tile layer.

9. The method of claim 1 , wherein in the generating, the tile data included in the selected user view video is analyzed in units of network abstraction layer (NAL) from the bitstream of the encoded tile layer, and a tile is included in the user view video is extracted to generate the tile bitstream.

10. An apparatus for streaming a multi-view 360 degree video, the apparatus comprising:
a video encoder encoding a multi-view video to a bitstream of a base layer and a bitstream of a tile layer constituted by at least one tile;
a user view tile selector selecting a tile included in a user view video in the encoded bitstream of the tile layer by using user view information received from a 360 degree video rendering apparatus, and video information of the multi-view video;
a tile extractor extracting tile data included in the selected user view video from the encoded bitstream of the tile layer, and generating a tile bitstream corresponding to the extracted tile data; and
a transmitter transmitting the encoded bitstream of the base layer and the generated tile bitstream to the 360 degree video rendering apparatus,
wherein the user view tile selector calculates a pixel warped to the user view video from the multi-view video by using a rotation matrix representing a rotation to the user view video from the multi-view video and a translation matrix representing a distance.

11. The apparatus of claim 10 , wherein the user view information includes rotation and position values of a head-mounted display worn by a user.

12. The apparatus of claim 10 , wherein the video information of the multi-view video includes a depth map of the multi-view video and a camera parameter of each video.

13. The apparatus of claim 10 , wherein the user view tile selector selects some pixels corresponding to each other in the multi-view video and the user view video, respectively to calculate whether a user view tile is included.

14. The apparatus of claim 13 , wherein a coordinate of a pixel to be used for calculating whether the user view tile is included is determined as a multiple of a Heuristic interval coefficient.

15. The apparatus of claim 10 , wherein the user view tile selector calculates a tile index of a pixel coordinate in the multi-view video corresponding to the calculated warped pixel by using a width and a height of the multi-view video and a width and a height of the tile.

16. The apparatus of claim 10 , wherein the user view tile selector adds, when the calculated warped pixel is included in a point set of the user view video, the tile index of the pixel coordinate in the multi-view video corresponding to the calculated warped pixel to an index set of the user view tile.

17. The apparatus of claim 16 , wherein the tile extractor extracts tile data corresponding to the added index set of the user view tile from the encoded bitstream of the tile layer.

18. The apparatus of claim 10 , wherein the tile extractor analyzes the tile data included in the selected user view video in units of network abstraction layer (NAL) from the bitstream of the encoded tile layer, and extracts a tile is included in the user view video to generate the tile bitstream.

19. A non-transitory computer readable storage medium for storing instructions, wherein when the instructions are executed by a processor, the instructions allow the processor to execute a method, the method comprising:
encoding a multi-view video to a bitstream of a base layer and a bitstream of a tile layer constituted by at least one tile;
selecting a tile included in a user view video in the encoded bitstream of the tile layer by using user view information received from a 360 degree video rendering apparatus, and video information of the multi-view video;
extracting tile data included in the selected user view video from the encoded bitstream of the tile layer, and generating a tile bitstream corresponding to the extracted tile data; and
transmitting the encoded bitstream of the base layer and the generated tile bitstream to the 360 degree video rendering apparatus, and
wherein in the selecting, a pixel warped to the user view video from the multi-view video is calculated by using a rotation matrix representing a rotation to the user view video from the multi-view video and a translation matrix representing a distance.

Assignees

• Research & Business Found Sungkyunkwan Univ

Inventors

• Jeong Jong Beom
• Lee Soon Bin
• Ryu Eun Seok

Classifications

• H04N21/234327

  by decomposing into layers, e.g. base layer and one or more enhancement layers · CPC title

• H04N13/178

  Metadata, e.g. disparity information · CPC title

• H04N13/366

  using viewer tracking · CPC title

• H04N19/174

  the region being a slice, e.g. a line of blocks or a group of blocks · CPC title

• H04N21/6587

  Control parameters, e.g. trick play commands, viewpoint selection · CPC title