Prioritizing tile-based virtual reality video streaming using adaptive rate allocation

Patent metadata

Field	Value
Publication number	US-10595069-B2
Application number	US-201715593050-A
Country	US
Kind code	B2
Filing date	May 11, 2017
Priority date	Dec 5, 2016
Publication date	Mar 17, 2020
Grant date	Mar 17, 2020

Abstract

The present disclosure includes methods and systems for streaming high-performance virtual reality video using adaptive rate allocation. In particular, an adaptive rate allocation system partitions a panorama video into segments or tiles and assigns priorities to each tile or segment based on input (e.g., a viewport of field-of-view) from a user client device. Further, the adaptive rate allocation system streams each tile or segment to the user client device according to the adaptive rate allocation, which maximizes bandwidth efficiency and video quality. In this manner, the adaptive rate allocation system delivers higher quality content to regions in the panorama video where a user is currently looking/most likely to look.

First claim

What is claimed is:

1. In a digital medium environment for providing a virtual reality experience, a computer-implemented method of streaming bandwidth-efficient virtual reality videos, the method comprising:
partitioning a panorama video into a plurality of tiles;
receiving viewport information about a viewport from a client device;
assigning, based on the viewport information, priorities to each tile of the plurality of tiles, wherein a first set of tiles of the plurality of tiles overlapping with the viewport is assigned a higher priority, and wherein a second set of tiles of the plurality of tiles adjacent to the viewport is assigned a lower priority;
determining a first surplus bandwidth amount by subtracting a minimum bit rate for the plurality of tiles from a bandwidth limit, the minimum bit rate corresponding to streaming the plurality of tiles of the panorama video at a minimum resolution;
determining a higher bit rate based on the first surplus bandwidth amount at which to stream the first set of tiles;
allocating the higher bit rate to the first set of tiles;
determining a second surplus bandwidth amount by subtracting the higher bit rate for the first set of tiles, the minimum bit rate for the second set of tiles, and the minimum bit rate for an unallocated set of tiles of the plurality of tiles from the bandwidth limit;
allocating a second bit rate, which is at least the minimum bit rate, to the second set of tiles based on the assigned lower priority and the second surplus bandwidth amount; and
streaming, to the client device, the first set of tiles at the higher bit rate and the second set of tiles at the second bit rate.

2. The method of claim 1 , further comprising employing a spatial relationship description to partition the panorama video into the plurality of tiles.

3. The method of claim 2 , wherein:
the panorama video comprises a 360-degree spherical video;
the plurality of tiles combines to form the 360-degree spherical video; and
partitioning the 360-degree spherical video into the plurality of tiles comprises generating one or more top tiles, one or more bottom tiles, and a plurality of middle body tiles.

4. The method of claim 3 , further comprising streaming the first set of tiles at the higher bit rate in parallel with streaming the second set of tiles at the at least minimum bit rate.

5. The method of claim 1 , further comprising:
detecting an object in one or more tiles from the second set of tiles adjacent to the viewport;
based on detecting the object, assigning the one or more tiles from the second set to the higher priority; and
allocating the higher bit rate to the one or more tiles from the second set of tiles.

6. The method of claim 1 , wherein the viewport information comprises position data indicating a location of a viewport within the panorama video.

7. The method of claim 1 , wherein the unallocated set of one or more tiles of the plurality of tiles neither overlap with the viewport nor are adjacent to the viewport.

8. The method of claim 1 , further comprising streaming the first set of tiles at the higher bit rate and the second set of tiles at the second bit rate without streaming duplicative data for a tile at different bit rates.

9. The method of claim 1 , further comprising:
determining that the second set of tiles assigned to the lower priority are streamable at the higher bit rate within the second surplus bandwidth amount; and
based on determining that the second set of tiles are streamable at the higher bit rate within the second surplus bandwidth amount, allocating the second set of tiles to the higher bit rate.

10. The method of claim 9 , further comprising:
determining that the second set of tiles are not streamable at the higher bit rate within the second surplus bandwidth amount;
based on determining that the second set of tiles are not streamable at the higher bit rate within the second surplus bandwidth amount, determining a next higher bit rate based on the second surplus bandwidth amount at which to stream the second set of tiles; and
allocating the next higher bit rate to the second set of tiles.

11. The method of claim 9 , further comprising:
determining that the second set of tiles are not streamable at the higher bit rate within the second surplus bandwidth amount;
allocating the minimum bit rate to the second set of tiles; and
allocating the minimum bit rate to the unallocated set of tiles.

12. The method of claim 10 , further comprising:
determining a remaining surplus bandwidth amount by subtracting the higher bit rate for the first set of tiles, the next higher bit rate for the second set of tiles, and the minimum bit rate for the unallocated set of tiles from the bandwidth limit;
determining that the unallocated set of tiles assigned to a lowest priority are streamable at the next higher bit rate with the remaining surplus bandwidth amount; and
based on determining that the unallocated set of tiles are streamable at the next higher bit rate with the remaining surplus bandwidth amount, allocating the unallocated set of tiles to the next higher bit rate.

13. The method of claim 6 , further comprising:
determining, based on the first surplus bandwidth amount, that the second set of tiles and the unallocated set of tiles are streamable at the higher bit rate; and
allocating the unallocated set of tiles to a lower bit rate based on a distance between the remaining set of tiles and the viewport satisfying a maximum viewing distance threshold.

14. A system for streaming virtual reality videos, the system comprising:
one or more memory devices comprising a virtual reality content repository; and
one or more server devices configured to cause the system to:
partition a panorama video into a plurality of tiles by dividing the panorama video into adjacent tiles;
receive viewport information for a viewport from a client device indicating a current viewport position within the panorama video;
assign, based on the viewport information, priorities to each tile of the plurality of tiles, wherein a first set of tiles of the plurality of tiles overlapping with the viewport is assigned a higher priority, and wherein a second set of tiles of the plurality of tiles adjacent to the viewport is assigned a lower priority;
determine a first surplus bandwidth amount by subtracting a minimum bit rate for the plurality of tiles from a bandwidth limit, the minimum bit rate corresponding to streaming the plurality of tiles of the panorama video at a minimum resolution;
determine a higher bit rate based on the first surplus bandwidth amount at which to stream the first set of tiles;
allocate the higher bit rate to the first set of tiles;
determine a second surplus bandwidth amount by subtracting the higher bit rate for the first set of tiles, the minimum bit rate for the second set of tiles, and the minimum bit rate for an unallocated set of tiles of the plurality of tiles from the bandwidth limit;
allocate a second bit rate, which is at least the minimum bit rate, to the second set of tiles based on the assigned lower priority and the second surplus bandwidth amount; and
stream, to the client device, the first set of tiles at the higher bit rate and the second set of tiles second bit rate.

15. The system of claim 14 , wherein the one or more server devices are further configured to cause the system to stream the first set of tiles at the higher bit rate and the second set of tiles at the at least minimum bit rate without streaming duplicative data for a tile at different bit rates.

16. The system of c

Assignees

• Adobe Inc

Inventors

• Swaminathan Viswanathan
• Hosseini Mohammad

Classifications

• G06T19/006

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• H04N21/64723

  Monitoring of network processes or resources, e.g. monitoring of network load (traffic related reporting in data switching networks H04L43/062) · CPC title

• H04N21/2662 Primary

  Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities · CPC title

• H04N21/8456 Primary

  by decomposing the content in the time domain, e.g. in time segments · CPC title

• H04L47/805

  QOS or priority aware · CPC title