Immersive virtual display

Patent metadata

Field	Value
Publication number	US-10643391-B2
Application number	US-201715713428-A
Country	US
Kind code	B2
Filing date	Sep 22, 2017
Priority date	Sep 23, 2016
Publication date	May 5, 2020
Grant date	May 5, 2020

Abstract

A VR system for vehicles that may implement methods that address problems with vehicles in motion that may result in motion sickness for passengers. The VR system may provide virtual views that match visual cues with the physical motions that a passenger experiences. The VR system may provide immersive VR experiences by replacing the view of the real world with virtual environments. Active vehicle systems and/or vehicle control systems may be integrated with the VR system to provide physical effects with the virtual experiences. The virtual environments may be altered to accommodate a passenger upon determining that the passenger is prone to or is exhibiting signs of motion sickness.

First claim

What is claimed is:

1. A system, comprising:
a virtual reality (VR) controller in a vehicle, wherein the VR controller comprises one or more processors and memory configured to:
obtain inputs from one or more sources in a vehicle following an actual route in a real environment at a location;
select a route in another real location, wherein selection is based on comparing turns and curves of the route in the other real location to turns and curves of the actual route, and wherein the route in the other location is selected based at least in part on the turns and curves of the route in the other real location at least partially matching the turns and curves of the actual route;
render frames including virtual content having a virtual environment comprising a simulation of the selected route in the other real location, wherein the virtual environment includes a virtual view that is rendered based on the selected route in the other real location, and wherein, in rendering the frames, the VR controller synchronizes motions and accelerations of the virtual content with motions and accelerations of the vehicle traveling along the actual route based on the inputs from the one or more sources in the vehicle;
send the frames to a VR device for display to a passenger in the vehicle, wherein the frames are rendered and displayed to provide the virtual view of the virtual environment that replaces a view of the actual route in the real environment from the vehicle via the VR device.

2. The system as recited in claim 1 , wherein the VR device includes a head-mounted device (HMD) that displays the frames to the passenger.

3. The system as recited in claim 1 , wherein the view of the virtual environment includes visual cues to indicate motion of the vehicle to the passenger.

4. The system as recited in claim 1 , wherein the one or more sources include:
one or more vehicle sensors, wherein the inputs from the vehicle sensors include one or more of depth map information from external vehicle sensors, localization information for the vehicle, video from one or more cameras on the vehicle, or motion and orientation information for the vehicle from an inertial-measurement unit (IMU) of the vehicle; and
one or more vehicle control systems, wherein the one or more vehicle control systems include one or more of a throttle control system, a brake control system, an active suspension control system, or a steering control system.

5. The system as recited in claim 1 , wherein the one or more sources include sources external to the vehicle that are accessed through wireless connections, wherein inputs from the external sources include one or more of world map data, 3D models of a local environment, information about objects or features in the local environment, video streams, or audio.

6. The system as recited in claim 1 , wherein the one or more sources include an inertial-measurement unit (IMU) of the VR device, wherein inputs from the IMU include motion and orientation information for the passenger's head.

7. The system as recited in claim 1 , wherein the VR controller is further configured to:
determine that the selected route does not exactly match the actual route; and
modify the simulation of the selected route based on differences between the actual route and the selected route.

8. The system as recited in claim 1 , wherein the VR controller is further configured to provide audio to the passenger through an audio system of the vehicle or through an audio system of the VR device.

9. The system as recited in claim 1 , wherein the VR controller is further configured to signal vehicle control systems to cause the vehicle control systems to react within constraints to provide physical effects to the passenger, wherein the vehicle control systems include one or more of a throttle control system, a brake control system, an active suspension control system, or a steering control system.

10. The system as recited in claim 1 , wherein the VR controller is further configured to signal an HVAC system of the vehicle to cause the HVAC system to change one or more of fan speed, temperature, or direction to provide physical effects to the passenger.

11. The system as recited in claim 1 , wherein the VR controller is further configured to signal an active seat to cause the seat to move or tilt to provide physical effects to the passenger.

12. The system as recited in claim 1 , wherein the VR controller is further configured to receive interactive inputs from the passenger and modify rendering of the frames including virtual reality content in response to the interactive inputs, wherein the interactive inputs include one or more of gestures, voice commands, or inputs to an input device.

13. The system as recited in claim 1 , wherein the VR controller is further configured to render avatars of two or more passengers in the vehicle in the virtual environment, wherein the two or more passengers participate in the virtual environment via respective VR devices.

14. The system as recited in claim 1 , wherein the VR controller is further configured to:
determine that the passenger is prone to or is exhibiting signs of motion sickness; and
upon determining that the passenger is prone to or is exhibiting signs of motion sickness, alter the virtual content according to one or more visual or audio accommodations to mitigate motion sickness of the passenger according to preference data for the passenger, wherein the one or more accommodations include:
displaying a ground plane beneath the passenger in the view of the virtual environment;
reducing visual cues that indicate motion of the vehicle in the virtual environment;
slowing the visual cues that indicate motion of the vehicle in the virtual environment so that there is a 1:1 ratio between motion in the virtual environment and motion in the real environment;
providing visual, audio, or haptic indications of upcoming vehicle maneuvers;
reducing the virtual content to provide a sparse visual environment;
placing an anchored object in the virtual environment that remains fixed relative to the real environment;
providing a visual stimulus at stops of the vehicle to counter visual motion aftereffects; or
providing audio output that blocks real world noise.

15. A method, comprising:
obtaining, by a virtual reality (VR) system in a vehicle, inputs from one or more sources in the vehicle following an actual route in a location;
selecting, by the VR system, a route in another real location, wherein selecting is based on comparing turns and curves of the route in the other real location to turns and curves of the actual route, and wherein the route in the other real location is selected based at least in part on the turns and curves of the route in the other real location at least partially matching the turns and curves of the actual route;
rendering, by the VR system, frames including virtual content having a virtual environment comprising a simulation of the route in the other real location, wherein the virtual environment includes a virtual view that is rendered based on the route in the other real location, and wherein, in rendering the frames, motions and accelerations of the virtual content are synchronized with motions and accelerations of the vehicle traveling along the actual route based on the inputs from the one or more sources in the vehicle; and
displaying, by the VR system, the frames to at least one passenger in the vehicle, wherein the displayed frames provide the virtual view of the virtual environment that replaces a view of the actual route in reality from the vehicle.

Assignees

• Apple Inc

Inventors

• Rober Mark B
• Cohen Sawyer I
• Kurz Daniel
• Holl Tobias
• Lyon Benjamin B
• Meier Peter Georg
• Riepling Jeffrey M
• Gerhard Holly

Classifications

• B60W10/18

  including control of braking systems · CPC title

• B60W10/04

  including control of propulsion units · CPC title

• G06F3/012

  Head tracking input arrangements · CPC title

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06T13/40

  of characters, e.g. humans, animals or virtual beings · CPC title