Virtual reality with interactive streaming video and likelihood-based foveation

Patent metadata

Field	Value
Publication number	US-10560680-B2
Application number	US-201715622836-A
Country	US
Kind code	B2
Filing date	Jun 14, 2017
Priority date	Jan 28, 2017
Publication date	Feb 11, 2020
Grant date	Feb 11, 2020

Abstract

A server may be configured to receive first pose data from a virtual reality device, generate second pose data, including a pose prediction, based at least on the first pose data, render a map representation based at least on the pose prediction, determine regions of the map representation based on a likelihood of view of each of the regions, perform foveation in each of the regions and generate an encoded frame based on the regions and a network parameter, and send the encoded frame and second pose data to the device. The encoding may associate each of the regions with a quality level and base the foveation on each region's associated quality level. The foveation may use different quantization parameters for each of the regions and/or use rendering foveation for each of regions. Network conditions such as latency, bandwidth, and/or jitter may be considered for bit allocation in the foveation.

First claim

What is claimed is:

1. A server comprising:
one or more processors, and memory in communication with the one or more processors, the memory comprising code that, when executed, causes the one or more processors to:
receive first pose data from a device;
generate second pose data, including a pose prediction, based at least on the first pose data;
render a map representation based at least on the pose prediction;
determine a plurality of regions of the map representation based on a likelihood of view of each of the plurality of regions;
perform foveation in each of the plurality of regions of the map representation and generate an encoded frame based on each of the plurality of regions and at least one network parameter; and,
send the encoded frame and second pose data to the device.

2. The server of claim 1 , wherein each of the plurality of regions of the map representation is associated with a quality level and the code is further executable to cause the one or more processors to perform the foveation for each of the plurality of regions based on each region's associated quality level.

3. The server of claim 1 , wherein the foveation includes performing encoding foveation using a different quantization parameter for each of the plurality of regions of the map representation.

4. The server of claim 1 , wherein the foveation includes performing rendering foveation for each of the plurality of regions of the map representation.

5. The server of claim 1 , wherein the at least one network parameter includes latency.

6. The server of claim 1 , wherein the at least one network parameter includes bandwidth.

7. The server of claim 1 , wherein the at least one network parameter includes jitter.

8. The server of claim 1 , wherein the code, when executed, further causes the one or more processors to perform foveation in each of the plurality of regions of the map representation and generate an encoded frame based on the likelihood of view of each of the plurality of regions and at least one network parameter.

9. The server of claim 8 , wherein the code causes the one or more processors to perform foveation by causing the processor to:
determine that the at least one network parameter has changed;
modify an allocation of bits between at least two of the plurality of regions based on the change in the at least one network parameter.

10. The server of claim 1 , wherein the map representation comprises a cube map representation.

11. A device comprising:
one or more processors, and memory in communication with the one or more processors, the memory comprising code that, when executed, causes the one or more processors to control the device to:
receive a first frame and first pose data from a server;
determine second pose data;
mesh warp the first frame using the first and second pose data to generate a first mesh warped frame;
display the first mesh warped frame;
determine third pose data;
determine if a second frame and fourth pose data have been received from the server; and,
if it is determined that a second frame has not been received:
mesh warp the first frame using the first and third pose data to generate a second mesh warped frame; and,
display the second mesh warped frame,

if it is determined that a second frame has been received:
mesh warp the second frame using the third and fourth pose data to generate a third mesh warped frame;
display the third mesh warped frame.

12. The device of claim 11 , wherein the code, when executed, further causes the one or more processors to control the device to send the third pose data to the server.

13. The device of claim 11 , wherein the frame comprises an image derived from a stereo cube map.

14. The device of claim 11 , wherein the device comprises a head mounted display and the pose data comprises a position of the device and a view orientation of a user of the device.

15. A method in a virtual reality system comprising:
receiving first pose data sent from a device at a server;
generating second pose data, including a pose prediction, based at least on the first pose data;
rendering a map representation based at least on the pose prediction;
determining a plurality of regions of the map representation based on a likelihood of view of each of the plurality of regions;
performing foveation in each of the plurality of regions of the map representation and generating an encoded frame based on each of the plurality of regions and at least one network parameter; and,
sending the encoded frame and second pose data from the server to the device.

16. The method of claim 15 , wherein each of the plurality of regions of the map representation is associated with a quality level and the performing the foveation for each of the plurality of regions comprises performing the foveation for each region based on its associated quality level and at least one network parameter.

17. The method of claim 15 , wherein the performing foveation comprises performing encoding foveation using a different quantization parameter for each of the plurality of regions of the map representation.

18. The method of claim 15 , further comprising:
receiving, at the device, the encoded frame and second pose data from the server;
determining third pose data;
decoding the encoded frame;
mesh warping the decoded frame using the second and third pose data to generate a mesh warped frame; and,
displaying the mesh warped frame at the device.

19. The method of claim 15 , wherein the performing foveation comprises performing rendering foveation for each of the plurality of regions of the map representation.

Assignees

• Microsoft Technology Licensing Llc

Inventors

• Horvitz Eric
• Cuervo Eduardo
• Chu David

Classifications

• H04N13/106

  Processing image signals (for multi-view video sequence encoding H04N19/597) · CPC title

• H04N21/440245

  the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment · CPC title

• H04N13/194

  Transmission of image signals · CPC title

• H04N21/234363

  by altering the spatial resolution, e.g. for clients with a lower screen resolution · CPC title

• H04N21/2402

  Monitoring of the downstream path of the transmission network, e.g. bandwidth available (traffic monitoring in data switching networks H04L43/00; monitoring data switching networks utilization H04L43/0876) · CPC title