Adaptive resolution of point cloud and viewpoint prediction for video streaming in computing environments

Patent metadata

Field	Value
Publication number	US-11178373-B2
Application number	US-201816050322-A
Country	US
Kind code	B2
Filing date	Jul 31, 2018
Priority date	Jul 31, 2018
Publication date	Nov 16, 2021
Grant date	Nov 16, 2021

Abstract

A mechanism is described for facilitating adaptive resolution and viewpoint-prediction for immersive media in computing environments. An apparatus of embodiments, as described herein, includes one or more processors to receive viewing positions associated with a user with respect to a display, and analyze relevance of media contents based on the viewing positions, where the media content includes immersive videos of scenes captured by one or more cameras. The one or more processors are further to predict portions of the media contents as relevant portions based on the viewing positions and transmit the relevant portions to be rendered and displayed.

First claim

What is claimed is:

1. An apparatus comprising:
one or more processors to:
receive viewing positions associated with a user with respect to a display;
analyze relevance of media contents based on the viewing positions, wherein the media content includes immersive videos of scenes captured by one or more cameras;
predict portions of the media contents as relevant portions based on the viewing positions, wherein the relevant portions are predicted by identifying one or more of future pose information relating to a head of the user with respect to the display based on the real-time pose information relating to the head of the user, and future physical displacement information of the user with respect to a real-time physical displacement of the user with respect to a real environment surrounding the user, wherein the future pose information includes one or more of future head positions or movements of the head with respect to the display, future user-visible objects or regions of the media contents, and future user-interested objects or regions of the media contents; and
transmit the relevant portions to be rendered and displayed.

2. The apparatus of claim 1 , wherein the one or more processors are further to analyze the relevance of the media contents by evaluating position information associated with the viewing positions, wherein the position information includes one or more of head pose information and physical displacement information, wherein the head pose information is based on movements or positions of the head of the user with respect to the display, and wherein the physical displacement information is based on physical displacements of the user in the real environment.

3. The apparatus of claim 1 , wherein the relevant portions of the media contents are predicted to be more likely to be viewed by the user based on one or more of the future head pose information and the future physical displacements, wherein the relevant portions include one or more of central objects or regions in a scene, distinct objects or regions, focused object or regions, and subject matter-pertinent objects or regions, wherein other portions of the media contents are predicted as irrelevant portions that are less likely to be viewed by the user based on one or more of the future head pose information and the future physical displacements, wherein the irrelevant portions include one or more of peripheral objects or regions in a scene, out-of-focus objects or regions, indistinct objects or regions, and subject matter-impertinent objects or regions.

4. The apparatus of claim 1 , wherein the one or more processors are further to encode the relevant portions of the media contents prior to transmitting the relevant portions, wherein the immersive video includes one or more of three degree-of-freedom+(3DoF+) video and six degree-of-freedom (6DoF) video, wherein the one or more processors comprise a graphics processor, wherein the graphics processor is co-located with an application processor on a common semiconductor package.

5. A method comprising:
receiving viewing positions associated with a user with respect to a display;
analyzing relevance of media contents based on the viewing positions, wherein the media content includes immersive videos of scenes captured by one or more cameras;
predicting portions of the media contents as relevant portions based on the viewing positions, wherein the relevant portions are predicted by identifying one or more of future pose information relating to a head of the user with respect to the display based on the real-time pose information relating to the head of the user, and future physical displacement information of the user with respect to a real-time physical displacement of the user with respect to a real environment surrounding the user, wherein the future head pose information includes one or more of future positions or movements of the head with respect to the display, future user-visible objects or regions of the media contents, and future user-interested objects or regions of the media contents; and
transmitting the relevant portions to be rendered and displayed.

6. The method of claim 5 , further comprising analyzing the relevance of the media contents by evaluating position information associated with the viewing positions, wherein the position information includes one or more of head pose information and physical displacement information, wherein the head pose information is based on movements or positions of the head of the user with respect to the display, and wherein the physical displacement information is based on physical displacements of the user in the real environment.

7. The method of claim 5 , wherein the relevant portions of the media contents are predicted to be more likely to be viewed by the user based on one or more of the future head pose information and the future physical displacements, wherein the relevant portions include one or more of central objects or regions in a scene, distinct objects or regions, focused object or regions, and subject matter-pertinent objects or regions, wherein other portions of the media contents are predicted as irrelevant portions that are less likely to be viewed by the user based on one or more of the future head pose information and the future physical displacements, wherein the irrelevant portions include one or more of peripheral objects or regions in a scene, out-of-focus objects or regions, indistinct objects or regions, and subject matter-impertinent objects or regions, wherein the one or more processors are further to encode the relevant portions of the media contents prior to transmitting the relevant portions, wherein the immersive video includes one or more of three degree-of-freedom+(3DoF+) video and six degree-of-freedom (6DoF) video, wherein the method is facilitated by one or more processors comprising a graphics processor coupled to an application processor and a memory, wherein the graphics processor is co-located with the application processor on a common semiconductor package.

8. At least one non-transitory computer-readable medium comprising instructions which, when executed, cause a computing device to perform operations comprising:
receiving viewing positions associated with a user with respect to a display;
analyzing relevance of media contents based on the viewing positions, wherein the media content includes immersive videos of scenes captured by one or more cameras;
predicting portions of the media contents as relevant portions based on the viewing positions, wherein the relevant portions are predicted by identifying one or more of future pose information relating to a head of the user with respect to the display based on the real-time pose information relating to the head of the user, and future physical displacement information of the user with respect to a real-time physical displacement of the user with respect to a real environment surrounding the user, wherein the future pose information includes one or more of future head positions or movements of the head with respect to the display, future user-visible objects or regions of the media contents, and future user-interested objects or regions of the media contents; and
transmitting the relevant portions to be rendered and displayed.

9. The non-transitory computer-readable medium of claim 8 , wherein the operations further comprise analyzing the relevance of the media contents by evaluating position information associated with the viewing positions, wherein the position information includes one or more of head pose information and physical displacement information, wherein the head pose information is based on movements or positions of a head of the user with respect to the display, and where

Assignees

• Intel Corp

Inventors

• Varerkar Mayuresh
• Baran Stanley
• Apodaca Michael
• Surti Prasoonkumar
• Kuwahara Atsuo
• Biswal Narayan
• Boyce Jill
• Chiu Yi-Jen
• Cilingir Gokcen
• Das Barnan
• Divekar Atul
• Potluri Srikanth
• Shah Nilesh
• Sharma Archie

Classifications

• H04N13/161

  Encoding, multiplexing or demultiplexing different image signal components (for multi-view video sequence encoding H04N19/597) · CPC title

• H04N13/366

  using viewer tracking · CPC title

• H04N13/271

  wherein the generated image signals comprise depth maps or disparity maps · CPC title

• H04N13/111 Primary

  Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation · CPC title

• G06N20/00

  Machine learning · CPC title