Versatile tile coding for multi-view video streaming

Patent metadata

Field	Value
Publication number	US-10972753-B1
Application number	US-201916569725-A
Country	US
Kind code	B1
Filing date	Sep 13, 2019
Priority date	Sep 13, 2019
Publication date	Apr 6, 2021
Grant date	Apr 6, 2021

Abstract

Techniques are disclosed for coding and delivering multi-view video in which the video is represented as a manifest file identifying a plurality of segments of the video available for download. The multi-view video may be partitioned spatially into a plurality of tiles that, in aggregate, encompass the entire spatial area of the video. The tiles are coded as segments contains coded video representing content contained within its respective tile. Tiles may be given different sizes based on saliency of the content within their respective regions. In this manner, tiles with high levels of interest may have relatively large spatial areas, which can lead to efficient coding in the presence of content motion.

First claim

We claim:

1. A video source device, comprising:
storage for coded video representing multi-view video, the coded video including a manifest file identifying a plurality of segments of the multi-view video available for download and network locations from which the segments may be downloaded, wherein
the multi view video is partitioned spatially into a plurality of tiles having sizes that are determined based on saliency of the content within their respective regions, and
each of the segments contains coded video representing content contained within a respective tile of the plurality of tiles.

2. The source device of claim 1 , wherein a tile corresponding to a saliency region of the multi-view video has a larger size than another tile that does not correspond to a saliency region.

3. The source device of claim 2 , wherein the saliency region corresponds to a region of interest identified from content of the multi-view video.

4. The source device of claim 2 , wherein the saliency region corresponds to a region of low complexity of the multi-view video.

5. The source device of claim 1 , wherein a first tile has a spatial area that overlaps with a spatial area of another tile.

6. The source device of claim 1 , wherein the tiles have respective spatial areas that do not overlap each other.

7. The source device of claim 1 , wherein the coded video includes segments coded at different tiers of service, each tier of service containing coded video of the multi-view video that is redundant but at a different quality to coded video contained within another tier of service.

8. The source device of claim 1 , wherein the coded video includes segments coded at different tiers of service, wherein the tiles of each tier of service, in aggregate, occupy an entire spatial area of the multi-view video.

9. The source device of claim 1 , wherein
the coded video includes segments coded at different tiers of service,
for at least one tier of service, plural sets of segments are provided, each set representing different partitioning of the multi-view video into tiles, and
the tiles of each partitioning, in aggregate, occupy an entire spatial area of the multi-view video.

10. The source device of claim 1 , wherein
the coded video includes segments coded at different tiers of service according to scalable coding in which
segments of a first tier of service are coded by base-layer coding, and
segments of a second tier of service are coded by enhancement-layer coding.

11. A video decoding method, comprising:
retrieving from a network a manifest file identifying a plurality of segments of a multi-view video available for download and tiles representing spatial areas of the multi-view video to which each segment corresponds, wherein the tiles are at sizes determined based on saliency of the content within their respective spatial areas,
selecting, from the tiles identified in the manifest file, segment(s) to be rendered,
retrieving from the network the selected segments according to network locations identified in the manifest file for the segments, and
decoding the selected segments.

12. The method of claim 11 , wherein the selecting comprises:
estimating a viewport location at a future time, and
selecting segments according to the estimated viewport location.

13. The method of claim 11 , wherein the selecting comprises, when the manifest file identifies different tiers of service for the multi-view video,
selecting a segment corresponding to an estimated viewport location at a first level of service, and
selecting another segment that does not correspond to the estimated viewport at a second, tier of service lower than the first tier of service.

14. The method of claim 11 , wherein the selecting comprises, when the manifest file identifies different tiers of service for the multi-view video,
selecting segments of a first tile corresponding to an estimated viewport location including a first segment containing a base-layer coded representation of the tile and at least one other segment containing an enhancement-layer coded representation of the tile, and
selecting a segment of a second tile that does not correspond to the estimated viewport containing a base-layer coded representation of the second tile.

15. The method of claim 11 , wherein the selecting comprises:
at a first time, predicting a viewport location at a future time, and downloading tiles of segment(s) associated with the predicted viewport location at a first level of coding quality, and
at a second time, re-predicting the viewport location at the future time and downloading tiles of segment(s) associated with the re-predicted viewport location at a second level of coding quality higher than the first level of coding quality.

16. The method of claim 11 , wherein the selecting comprises, when the manifest file identifies different tiers of service for the multi-view video,
estimating a first viewport location at a prefetch time, and selecting first quality representation of segments according to the first estimated viewport location, and
estimating a second viewport location at a later time, and selecting a second quality representation of segments according to the second estimated viewport location.

17. The method of claim 11 , wherein, when the manifest file identifies different tiers of service for the multi-view video:
the coded video includes segments coded at different tiers of service,
for at least one tier of service, plural sets of segments are provided, each set representing different partitioning of the multi-view video into tiles, and
the tiles of each partitioning, in aggregate, occupy an entire spatial area of the multi-view video.

18. The method of claim 11 , wherein a first tile has a spatial area that overlaps with a spatial area of another tile.

19. The method of claim 11 , wherein the tiles have respective spatial areas that do not overlap each other.

20. Non-transitory computer readable medium containing program instructions that, when executed by a player device, cause the device to perform a method, comprising:
retrieving from a network a manifest file identifying a plurality of segments of a multi-view video available for download and tiles representing spatial areas of the multi-view video to which each segment corresponds, wherein the tiles are at sizes determined based on saliency of the content within their respective spatial areas,
selecting, from the tiles identified in the manifest file, segment(s) to be rendered, retrieving from the network the selected segments according to network locations identified in the manifest file for the segments, and
decoding the selected segments.

21. The medium of claim 20 , wherein the selecting comprises:
estimating a viewport location at a future time, and
selecting segments according to the estimated viewport location.

22. The medium of claim 20 , wherein the selecting comprises, when the manifest file identifies different tiers of service for the multi-view video,
selecting a segment corresponding to an estimated viewport location at a first level of service, and
selecting another segment that does not correspond to the estimated viewport at a second, tier of service lower than the first tier of service.

23. The medium of claim 20 , wherein the selecting comprises, when the manifest file identifies different tiers of service for the multi-view video,
selecting segments of a first tile corresponding to an

Assignees

• Apple Inc

Inventors

• Duanmu Fanyi
• Asbun Eduardo
• Zhou Xiaosong
• Xin Jun
• Wu Hsi-Jung
• Su John
• Gehani Samir
• Flick christopher
• Sahoo Shalini

Classifications

• H04N21/44008

  involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59) · CPC title

• H04N21/8456

  by decomposing the content in the time domain, e.g. in time segments · CPC title

• H04N21/431 Primary

  Generation of visual interfaces {for content selection or interaction}; Content or additional data rendering · CPC title

• H04N21/816

  involving special video data, e.g 3D video · CPC title

• H04N21/23439

  for generating different versions · CPC title