Augmented virtual display

Patent metadata

Field	Value
Publication number	US-10482669-B2
Application number	US-201715713421-A
Country	US
Kind code	B2
Filing date	Sep 22, 2017
Priority date	Sep 23, 2016
Publication date	Nov 19, 2019
Grant date	Nov 19, 2019

Abstract

A VR system for vehicles that may implement methods that address problems with vehicles in motion that may result in motion sickness for passengers. The VR system may provide augmented or virtual views that match visual cues with the physical motions that a passenger experiences. The VR system may project virtual content so that the content appears as a distant object stabilized or fixed in the external environment. The VR system may aid in productivity, as passengers may perform work while riding in the vehicle without experiencing motion sickness. In addition, the VR system may provide enhanced virtual experiences to passengers in moving vehicles by matching accelerations and motions of the vehicle to accelerations and motions in the virtual experiences.

First claim

What is claimed is:

1. A system, comprising:
a virtual reality (VR) controller in a vehicle, wherein the VR controller is configured to:
obtain inputs from one or more sources in the vehicle;
obtain display screen content from a user device of a passenger in the vehicle;
render virtual content including the display screen content from the user device based at least in part on the inputs from the one or more sources in the vehicle;
alter the virtual content according to one or more visual or audio accommodations to mitigate motion sickness of the passenger according to preference data for the passenger; and
send the virtual content to a VR device for display in a 3D virtual view to the passenger in the vehicle, wherein the display screen content from the user device is rendered and displayed to appear to the passenger to be at a fixed distance in front of the vehicle.

2. The system as recited in claim 1 , wherein the VR device includes:
a head-mounted device (HMD) worn by the passenger that displays the virtual content for viewing by the passenger; or
a projector that projects the virtual content to a window of the vehicle for viewing by the passenger.

3. The system as recited in claim 1 , wherein the user device comprises one or more of a notebook computer, a mobile device, a smartphone, or a tablet.

4. The system as recited in claim 1 , wherein the virtual content is displayed in a view of a real world scene in front of the vehicle to provide an augmented or mixed reality view to the passenger.

5. The system as recited in claim 1 , wherein the virtual content including the display screen content represents a virtual environment that replaces a view of a real world scene in front of the vehicle with a virtual reality view, wherein the virtual reality view includes visual cues that indicate motion of the vehicle in the virtual environment to the passenger.

6. The system as recited in claim 5 , wherein the VR controller is further configured to:
determine that the passenger is prone to or is exhibiting signs of motion sickness; and
upon determining that the passenger is prone to or is exhibiting signs of motion sickness, alter the virtual environment according to one or more accommodations to mitigate motion sickness.

7. The system as recited in claim 6 , wherein the VR controller is configured to determine that the passenger is prone to motion sickness according to the preference data for the passenger or input from the passenger.

8. The system as recited in claim 6 , wherein the VR controller is configured to determine that the passenger is exhibiting signs of motion sickness according to passenger sensor data obtained from one or more sensors that monitor the passenger.

9. The system as recited in claim 6 , wherein the one or more accommodations include:
displaying a ground plane beneath the passenger in the virtual reality view;
reducing a number of visual cues that indicate motion of the vehicle in the virtual environment;
slowing the visual cues that indicate motion of the vehicle in the virtual environment so that there is a 1:1 ratio between motion in the virtual environment and motion in the real environment;
providing visual, audio, or haptic indications of upcoming vehicle maneuvers;
reducing the virtual content displayed in the virtual reality view to provide a sparse visual environment;
placing an anchored object in the virtual environment that remains fixed relative to the real world;
providing a visual stimulus at stops of the vehicle to counter visual motion aftereffects; or
providing audio output that blocks real world noise.

10. The system as recited in claim 1 , wherein the one or more sources include:
one or more vehicle sensors, wherein the inputs from the vehicle sensors include one or more of depth map information from external vehicle sensors, localization information for the vehicle, video from one or more cameras on the vehicle, or motion and orientation information for the vehicle from an inertial-measurement unit (IMU) of the vehicle; and
one or more vehicle control systems, wherein the one or more vehicle control systems include one or more of a throttle control system, a brake control system, an active suspension control system, or a steering control system.

11. The system as recited in claim 1 , wherein the one or more sources include sources external to the vehicle that are accessed through wireless connections, wherein inputs from the external sources include one or more of world map data, 3D models of a local environment, information about objects or features in the local environment, video streams, or audio.

12. The system as recited in claim 1 , wherein the one or more sources include an inertial-measurement unit (IMU) of the VR device, wherein inputs from the IMU include motion and orientation information for the passenger's head.

13. The system as recited in claim 1 , wherein the vehicle is an autonomous vehicle.

14. The system as recited in claim 1 , wherein, in rendering the virtual content, the VR controller synchronizes motions and accelerations of the virtual content with motions and accelerations of the vehicle based on the inputs from the one or more sources in the vehicle.

15. A method, comprising:
obtaining, by a virtual reality (VR) system in a vehicle, inputs from one or more sources in the vehicle;
obtaining, by the VR system, display screen content from a user device of a passenger in the vehicle;
rendering, by the VR system, virtual content including the display screen content from the user device based at least in part on the inputs from the one or more sources in the vehicle;
altering the virtual content according to one or more visual or audio accommodations to mitigate motion sickness of the passenger according to preference data for the passenger; and
displaying, by the VR system, the virtual content in a 3D virtual view to a passenger in the vehicle, wherein the display screen content appears to the passenger to be at a fixed distance in front of the vehicle.

16. The method as recited in claim 15 , wherein the VR system includes a VR controller and a VR device, wherein the VR device is one of a head-mounted device (HMD) configured to be worn by the passenger and to display or project the virtual content for viewing by the passenger or a projector configured to project the virtual content to a window of the vehicle for viewing by the passenger.

17. The method as recited in claim 15 , wherein the virtual content is displayed in a view of a real world scene in front of the vehicle to provide an augmented or mixed reality view to the passenger.

18. The method as recited in claim 15 , wherein the virtual content including the display screen content represents a virtual environment that replaces a view of a real world scene in front of the vehicle with a virtual reality view, wherein the virtual reality view includes visual cues to indicate motion of the vehicle in the virtual environment to the passenger.

19. The method as recited in claim 15 , further comprising:
determining that the passenger is prone to or is exhibiting signs of motion sickness, wherein altering the virtual content according to the one or more visual or audio accommodations to mitigate motion sickness is performed upon determining that the passenger is prone to or is exhibiting signs of motion sickness.

20. The method as recited in claim 15 , wherein rendering the virtual content comprises synchronizing motions and accelerations of the virtual content with motions a

Assignees

• Apple Inc

Inventors

• Rober Mark B
• Cohen Sawyer I
• Kurz Daniel
• Holl Tobias
• Lyon Benjamin B
• Meier Peter Georg
• Riepling Jeffrey M
• Gerhard Holly

Classifications

• G06F3/011

  Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00) · CPC title

• G06T13/40

  of characters, e.g. humans, animals or virtual beings · CPC title

• G06T19/006 Primary

  Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00) · CPC title

• B60W10/04

  including control of propulsion units · CPC title

• B60R2300/207

  using multi-purpose displays, e.g. camera image and navigation or video on same display · CPC title