Method for streaming ultra high definition panoramic videos

Patent metadata

Field	Value
Publication number	US-11159776-B2
Application number	US-201916713172-A
Country	US
Kind code	B2
Filing date	Dec 13, 2019
Priority date	Aug 16, 2019
Publication date	Oct 26, 2021
Grant date	Oct 26, 2021

Abstract

Aspects of the subject disclosure may include, for example, a device in which a processing system receives from a multimedia content server a plurality of content streams; each of the content streams includes a portion of an original content stream. The processing system transcodes the plurality of content streams based on a viewport prediction to produce a plurality of viewport streams; the viewport prediction is based at least in part on a visibility map associated with a viewer of the content. The processing system delivers each of the viewport streams to a client device associated with the viewer; each of the viewport streams is buffered at the client device in a separate buffer. Other embodiments are disclosed.

First claim

What is claimed is:

1. A device, comprising:
a processing system including a processor; and
a memory that stores executable instructions that, when executed by the processing system, facilitate performance of operations, the operations comprising:
receiving from a multimedia content server a plurality of content streams, each of the content streams comprising a portion of an original content stream comprising a panoramic video;
receiving from the server a panoramic stream in addition to the plurality of content streams, the panoramic stream corresponding to the panoramic video and having a resolution less than that of the plurality of content streams;
transcoding the plurality of content streams based on a viewport prediction of a viewport to produce a plurality of viewport streams, the viewport prediction based at least in part on a visibility map associated with a viewer of the content;
delivering the plurality of viewport streams to a client device associated with the viewer, wherein each of the plurality of viewport streams is buffered at the client device in a separate buffer; and
forwarding the panoramic stream to the client device without transcoding the panoramic stream and without regard to the viewport, wherein the panoramic stream is buffered at the client device separate from the plurality of viewport streams.

2. The device of claim 1 , wherein the original content stream is divided by the server to form the plurality of content streams.

3. The device of claim 1 , wherein the original content stream comprises a 16K panoramic video stream, and wherein each of the plurality of content streams comprises a 8K×8K video stream.

4. The device of claim 1 , wherein the processing system comprises an edge cloudlet, and wherein the transcoding is performed in real time by a plurality of processors.

5. The device of claim 1 , wherein the viewport prediction is based at least in part on a real-time trajectory of the viewer.

6. The device of claim 1 , wherein the viewport prediction is based at least in part on a long short-term memory (LSTM) deep learning model.

7. The device of claim 1 , wherein the plurality of viewport streams are delivered to the client device over a network.

8. The device of claim 7 , wherein the operations further comprise adjusting a quality of the plurality of viewport streams based on a network condition.

9. The device of claim 1 , wherein video content is rendered to the viewer by presenting a portion of the panoramic stream in the predicted viewport and subsequently overwriting the portion of the panoramic stream by presenting a portion of at least one of the plurality of viewport streams in the predicted viewport.

10. A non-transitory machine-readable medium comprising executable instructions that, when executed by a processing system including a processor, facilitate performance of operations, the operations comprising:
receiving from a multimedia content server a plurality of content streams, each of the content streams comprising a portion of an original content stream comprising a panoramic video;
receiving from the server a panoramic stream in addition to the plurality of content streams, the panoramic stream corresponding to the panoramic video and having a resolution less than that of the plurality of content streams;
transcoding the plurality of content streams based on a viewport prediction of a viewport to produce a plurality of viewport streams, the viewport prediction based at least in part on a visibility map associated with a viewer of the content, the transcoding performed in real time by a plurality of processors;
delivering the plurality of viewport streams to a client device associated with the viewer,
wherein each of the plurality of viewport streams is buffered at the client device in a separate buffer; and
forwarding the panoramic stream to the client device without transcoding the panoramic stream and without regard to the viewport, wherein the panoramic stream is buffered at the client device separate from the plurality of viewport streams.

11. The non-transitory machine-readable medium of claim 10 , wherein the original content stream is divided by the server to form the plurality of content streams.

12. The non-transitory machine-readable medium of claim 10 , wherein the original content stream comprises a 16K panoramic video stream, and wherein each of the plurality of content streams comprises a 8K×8K video stream.

13. The non-transitory machine-readable medium of claim 10 , wherein the processing system comprises an edge cloudlet.

14. The non-transitory machine-readable medium of claim 10 , wherein the viewport prediction is based at least in part on a real-time trajectory of the viewer.

15. The non-transitory machine-readable medium of claim 10 , wherein the plurality of viewport streams are delivered to the client device over a network, and wherein the operations further comprise adjusting a quality of the plurality of viewport streams based on a network condition.

16. A method comprising:
receiving, by a processing system of an edge cloudlet including a processor, from a multimedia content server a plurality of content streams, each of the content streams comprising a portion of an original content stream;
receiving, by the processing system, from the server a panoramic stream in addition to the plurality of content streams, the panoramic stream corresponding to the panoramic video and having a resolution less than that of the plurality of content streams;
transcoding, by the processing system, the plurality of content streams based on a viewport prediction of a viewport to produce a plurality of viewport streams, the viewport prediction based at least in part on a visibility map associated with a viewer of the content;
delivering, by the processing system, the plurality of viewport streams to a client device associated with the viewer,
wherein each of the plurality of viewport streams is buffered at the client device in a separate buffer; and
forwarding the panoramic stream to the client device without transcoding the panoramic stream and without regard to the viewport, wherein the panoramic stream is buffered at the client device separate from the plurality of viewport streams.

17. The method of claim 16 , wherein the original content stream is divided by the server to form the plurality of content streams.

18. The method of claim 16 , wherein the original content stream comprises a 16K panoramic video stream, and wherein each of the plurality of content streams comprises a 8K×8K video stream.

19. The method of claim 16 , wherein the edge cloudlet comprises a plurality of processors.

20. The method of claim 16 , wherein the plurality of viewport streams are delivered to the client device over a network, and further comprising adjusting, by the processing system, a quality of the plurality of viewport streams based on a network condition.

Assignees

• At & T Ip I Lp
• Univ Hong Kong Science & Tech

Inventors

• Han Bo
• Zhang Wenxiao
• Hui Pan
• Xu Tan
• Ip Cheuk Yiu

Classifications

• H04N21/64792 Primary

  Controlling the complexity of the content stream, e.g. by dropping packets (intermediate media network packet handling H04L65/765; proxy provisioning conversion or adaptation for reducing the amount or size of exchanged application data H04L67/5651; negotiation of resources in wireless networks H04W28/16) · CPC title

• H04N21/21805

  enabling multiple viewpoints, e.g. using a plurality of cameras · CPC title

• H04N21/23439

  for generating different versions · CPC title

• H04N21/8456

  by decomposing the content in the time domain, e.g. in time segments · CPC title

• H04N13/282

  for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems · CPC title