System and method of predicting field of view for immersive video streaming

Patent metadata

Field	Value
Publication number	US-11282283-B2
Application number	US-202016933494-A
Country	US
Kind code	B2
Filing date	Jul 20, 2020
Priority date	Oct 2, 2017
Publication date	Mar 22, 2022
Grant date	Mar 22, 2022

Abstract

Aspects of the subject disclosure may include, for example, a process that analyzes image content of an immersive video frame and identifies a first group of segments of the immersive video frame based on the analysis. The segments collectively span an entire space portrayed by the immersive video frame and the first group of segments spans less than the entire space. A size of a display region of an immersive video viewer is determined that spans less than the entire space and a second group of segments is determined based on the first group of segments and the size of the display region. Transport of the second group of segments, spanning less than the entire space, is facilitated via a communication network to the immersive video viewer for presentation, without requiring transmission of all of the segments. Other embodiments are disclosed.

First claim

What is claimed is:

1. A method, comprising:
determining, by a processing system including a processor, a field-of-view of an immersive video viewer, wherein the field-of-view spans less than an entire image space portrayed by a first immersive video frame of a sequence of immersive video frames;
identifying, by the processing system, a second group of spatial segments of a plurality of spatial segments of the sequence of immersive video frames based on a selection of a first group of spatial segments and the determining of the field-of-view, wherein the plurality of spatial segments collectively span an entire temporal image space portrayed by the sequence of immersive video frames, and wherein the first group of spatial segments and the second group of spatial segments span less than the entire image space portrayed by the first immersive video frame; and
facilitating, by the processing system, transport of the second group of spatial segments via a communication network to the immersive video viewer for presentation at the immersive video viewer, without requiring transmission of all the plurality of spatial segments.

2. The method of claim 1 , further comprises:
applying, by the processing system, image processing to image content of the first immersive video frame; and
identifying, by the processing system, an object portrayed in the image content based on the image processing.

3. The method of claim 2 , further comprising associating, by the processing system, segments of the plurality of spatial segments that comprise a portion of the object, with the first group of spatial segments.

4. The method of claim 1 , further comprises:
obtaining heat-map data indicating viewing frequencies of sub-regions of the first immersive video frame of the sequence of immersive video frames.

5. The method of claim 4 , wherein the identifying of the second group of spatial segments of the plurality of spatial segments comprises predicting a position of the field-of-view within the first immersive video frame based on the heat-map data.

6. The method of claim 1 , wherein the identifying of the second group of spatial segments further comprises:
determining, by the processing system, an overlap of the field-of-view and the first group of spatial segments and the determining of the field-of-view, wherein the second group of spatial segments spans the field-of-view in its entirety, while spanning less than the entire image space portrayed by the first immersive video frame.

7. The method of claim 1 , further comprising:
selecting, by the processing system, a third group of spatial segments of the plurality of spatial segments based on second group of spatial segments, wherein the selecting of the third group of spatial segments is not based on analysis of image content of a second immersive video frame of the sequence of immersive video frames.

8. The method of claim 1 , wherein the plurality of spatial segments is non-overlapping.

9. The method of claim 1 , wherein the field-of-view is based on an estimate of an orientation of the immersive video viewer, the method further comprising:
predicting, by the processing system, the orientation of the immersive video viewer to obtain a prediction, wherein the estimate of the orientation of the immersive video viewer is based on the prediction of the orientation of the immersive video viewer.

10. The method of claim 1 , further comprising:
identifying, by the processing system, an identity of a user of the immersive video viewer; and
determining, by the processing system, a viewer characteristic based on the identity of the user of the immersive video viewer, wherein the identifying of the second group of spatial segments of the plurality of spatial segments is further based on the viewer characteristic.

11. The method of claim 10 , wherein the determining of the field-of-view of the first immersive video frame is based on the viewer characteristic.

12. The method of claim 10 , wherein the determining of the viewer characteristic further comprises: accessing, by the processing system, a user profile associated with the identity of the user.

13. A non-transitory, machine-readable storage medium, comprising executable instructions that, when executed by a processing system including a processor, facilitate performance of operations, the operations comprising:
determining a field-of-view of an immersive video viewer, wherein the field-of-view spans less than an entire image space portrayed by a first immersive video frame of a sequence of immersive video frames;
identifying a second group of segments of a plurality of segments of the sequence of immersive video frames based on a selection of a first group of spatial segments and the determining of the field-of-view, wherein the plurality of segments collectively span an entire temporal image space portrayed by the sequence of immersive video frames, and wherein the first group of segments and the second group of segments span less than the entire image space portrayed by the first immersive video frame; and
facilitating transport of the second group of segments via a communication network to the immersive video viewer for presentation at the immersive video viewer, without requiring transmission of all the plurality of segments.

14. The non-transitory, machine-readable storage medium of claim 13 , wherein the operations further comprise:
applying, by the processing system, image processing to image content of the first immersive video frame; and
identifying, by the processing system, an object portrayed in the image content based on the image processing.

15. The non-transitory, machine-readable storage medium of claim 14 , wherein the selecting of the first group of segments further comprises associating, by the processing system, segments of the plurality of segments that comprise a portion of the object, with the first group of segments.

16. The non-transitory, machine-readable storage medium of claim 13 , wherein the plurality of segments is non-overlapping.

17. The non-transitory, machine-readable storage medium of claim 13 , wherein the determining of the second group of segments further comprises:
determining, by the processing system, an overlap of the field-of-view and the first group of segments and the determining of the field-of-view, wherein the second group of segments spans the field-of-view in its entirety, while spanning less than the entire image space portrayed by the first immersive video frame.

18. A device, comprising:
a processing system including a processor; and
a memory that stores executable instructions that, when executed by the processing system, facilitate performance of operations, the operations comprising:
determining a field-of-view of an immersive video viewer, wherein the field-of-view spans less than an entire image space portrayed by an immersive video frame of a sequence of immersive video frames;
identifying a second group of segments of a plurality of segments of the sequence of immersive video frames based on a selection of a first group of spatial segments and the second group of segments span less than the entire image space portrayed by the immersive video frame; and
facilitating transport of the second group of segments via a communication network to the immersive video viewer for presentation at the immersive video viewer, without requiring transmission of all the plurality of segments.

19. The device of claim 18 , wherein the operations further comprise:
applying image processing to image

Assignees

• At & T Ip I Lp

Inventors

• Han Bo
• Gopalakrishnan Vijay
• Pair Jackson Jarrell
• Sharma Vikash

Classifications

• H04N21/21805 Primary

  enabling multiple viewpoints, e.g. using a plurality of cameras · CPC title

• H04L67/131

  Protocols for games, networked simulations or virtual reality · CPC title

• H04N21/816

  involving special video data, e.g 3D video · CPC title

• H04N21/234

  Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs · CPC title

• G06F3/04815

  Interaction with a metaphor-based environment or interaction object displayed as three-dimensional [3D], e.g. changing the user viewpoint with respect to the environment or object · CPC title