Virtual reality experience for a vehicle

Patent metadata

Field	Value
Publication number	US-9922466-B2
Application number	US-201615230053-A
Country	US
Kind code	B2
Filing date	Aug 5, 2016
Priority date	Aug 5, 2016
Publication date	Mar 20, 2018
Grant date	Mar 20, 2018

Abstract

A virtual reality system is disclosed that provides autonomous vehicle (AV) sensor data to applications such as games and augmented reality overlays to enhance experiences for riders in the autonomous vehicle. Virtual reality headsets offer users unique and interesting experiences, but when used in a changing environment such as a moving vehicle, external stimuli can impair the virtual reality experience. AV sensors can predict these stimuli so that applications can take measures to reduce their impacts on virtual reality experiences. In addition, sensors can include cameras that send live video feeds to virtual reality devices to render improved views of the environment around the AV and of landmarks in a city. Furthermore, virtual reality devices can take advantage of the AV's computing resources in order to offer better performance and more features to applications.

First claim

What is claimed is:

1. An autonomous vehicle (AV) comprising:
a sensor system to generate sensor data corresponding to a situational environment of the AV;
a data processing system to determine that values for the sensor data exceed one or more programmed thresholds for a virtual reality experience;
a control system to utilize the sensor data to autonomously control the AV along a current route;
one or more computing resources shared between a virtual reality system and other systems of the AV, including at least one of the sensor system, the data processing system, and the control system; and
the virtual reality system to:
provide the sensor data to a virtual reality application, executed on a second computing device, based on the values for the sensor data; and
manage utilization of the one or more computing resources between the second computing device and the other systems of the AV, wherein managing utilization includes (i) monitoring usage of the one or more computing resources by the other systems of the AV, (ii) determining that excess resources are available, and (iii) making the excess resources available to the virtual reality application to render the virtual reality experience in a higher fidelity or resolution and display the virtual reality experience to a rider in the AV.

2. The autonomous vehicle of claim 1 , wherein the virtual reality application accesses the sensor data through an application programming interface provided by the virtual reality system.

3. The autonomous vehicle of claim 1 , wherein the sensor system tracks a head position and view direction of the rider in the AV and provides the head position and view direction to the virtual reality application.

4. The autonomous vehicle of claim 1 , wherein the sensor system tracks a hand position and body position for one or more riders in the AV and provides the hand position and the body position for each of the riders to the virtual reality application.

5. The autonomous vehicle of claim 1 , wherein the virtual reality system retrieves content from one or more databases over a network based on the sensor data and provides the retrieved content to the virtual reality application.

6. The autonomous vehicle of claim 1 , wherein the data processing system retrieves route data corresponding to the current route from one or more databases over a network, and the virtual reality system provides the route data to the virtual reality application.

7. The autonomous vehicle of claim 1 , wherein the sensor system tracks one or more possessions associated with the rider in the AV and warns the rider when a different rider in the AV accesses one of the possessions.

8. The autonomous vehicle of claim 1 , wherein the virtual reality system provides a 3D model of the interior of the AV to the virtual reality application.

9. The autonomous vehicle of claim 8 , wherein the 3D model of the interior of the AV provided to the virtual reality application includes a representation of haptic textures of surfaces in the interior of the AV.

10. The autonomous vehicle of claim 1 , further comprising a signal emitter to communicate with one or more remote sensors in the situational environment, wherein the one or more remote sensors provide the AV with further sensor data.

11. A virtual reality system for an autonomous vehicle (AV) comprising:
one or more computing resources, including one or more processors, shared between the virtual reality system and other systems of the AV, including at least one of a sensor system, a data processing system, and a control system for the AV that utilizes the sensor data to autonomously control the AV along a current route; and
one or more memory resources storing instructions that, when executed by the one or more processors, cause the virtual reality system to:
retrieve, from the data processing system of the AV, sensor data corresponding to a situational environment of the AV, wherein the data processing system determines that values for the sensor data exceed one or more programmed thresholds for a virtual reality experience;
provide the sensor data to a virtual reality application, executed on a second computing device, based on the values for the sensor data; and
manage utilization of the one or more computing resources between the second computing device and the other systems of the AV, wherein managing utilization includes (i) monitoring usage of the one or more computing resources by the other systems of the AV, (ii) determining that excess resources are available, and (iii) making the excess resources available to the virtual reality application to render the virtual reality experience in a higher fidelity or resolution and display the virtual reality experience to a rider in the AV.

12. The virtual reality system of claim 11 , wherein the virtual reality application accesses the sensor data through an application programming interface provided by the virtual reality system.

13. The virtual reality system of claim 11 , wherein the sensor system tracks a head position and view direction of the rider in the AV and provides the head position and view direction to the virtual reality application.

14. The virtual reality system of claim 11 , wherein the sensor system tracks a hand position and body position of the rider in the AV and provides the hand position and the body position to the virtual reality application.

15. The virtual reality system of claim 11 , wherein the virtual reality system retrieves content from one or more databases over a network based on the sensor data and provides the retrieved content to the virtual reality application.

16. The virtual reality system of claim 11 , wherein the data processing system retrieves route data corresponding to the current route from one or more databases over a network, and the virtual reality system provides the route data to the virtual reality application.

17. A non-transitory computer-readable medium storing instructions for a virtual reality system of an autonomous vehicle (AV), wherein the instructions when executed by one or more processors, cause the one or more processors to:
retrieve, from a data processing system of the AV, sensor data corresponding to a situational environment of the AV, wherein the data processing system determines that values for the sensor data exceed one or more programmed thresholds for a virtual reality experience;
provide the sensor data to a virtual reality application, executed on a second computing device, based on the values for the sensor data; and
manage utilization of one or more computing resources, shared between the virtual reality system and other systems of the AV, including at least one of a sensor system, the data processing system, and a control system for the AV that utilizes the sensor data to autonomously control the AV along a current route, between the second computing device and the other systems of the AV, wherein managing utilization includes (i) monitoring usage of the one or more computing resources by the other systems of the AV, (ii) determining that excess resources are available, and (iii) making the excess resources available to the virtual reality application to render the virtual reality experience in a higher fidelity or resolution and display the virtual reality experience to a rider in the AV.

Assignees

• Uber Technologies Inc

Inventors

• Donnelly Richard
• Bradley David Mcallister

Classifications

• G06F3/012 Primary

  Head tracking input arrangements · CPC title

• G06F3/016

  Input arrangements with force or tactile feedback as computer generated output to the user · CPC title

• B60R1/00

  Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles · CPC title

• B60R2300/207

  using multi-purpose displays, e.g. camera image and navigation or video on same display · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title