Method and apparatus for visualizing future events for passengers of autonomous vehicles

Patent metadata

Field	Value
Publication number	US-2020041997-A1
Application number	US-201816054752-A
Country	US
Kind code	A1
Filing date	Aug 3, 2018
Priority date	Aug 3, 2018
Publication date	Feb 6, 2020
Grant date	—

Abstract

An approach is provided for visualizing future events to a passenger of a vehicle. The approach involves collecting sensor data, contextual data, or a combination thereof during an operation of a vehicle. The approach also involves processing the sensor data, the contextual data, or a combination thereof to predict a future event that is expected to occur proximate to the vehicle. The approach further involves generating a simulation of the future event. The approach further involves presenting the simulation in a user interface

First claim

1 . A computer-implemented method comprising:
collecting, by one or more processors, sensor data, contextual data, or a combination thereof during an operation of a vehicle; processing, by the one or more processors, the sensor data, the contextual data, or a combination thereof to predict a future event that is expected to occur proximate to the vehicle, wherein the future event involves a change of context of at least one object external to the vehicle; generating a simulation of the future event; and before the future event occurs, performing, by the one or more processors, one or more of: (i) causing one or more displays to display the simulation of the future event in the vehicle, (ii) causing one or more speakers to output audio in the vehicle in accordance with the simulation of the future event, or (iii) causing one or more physical actuators in the vehicle to actuate in accordance with the simulation of the future event.

2 . The method of claim 1 , further comprising:
determining that the object will be proximate to the vehicle based on a relevance level of the object to the user in the vehicle, wherein the change of the context of the object is presented via the one or more displays, the one or more speakers, the one or more physical actuators, or a combination thereof.

3 . The method of claim 2 , wherein the object is presented in the vehicle while physically located outside of a field of view of a user in the vehicle, and the future event includes an autonomous maneuver that the vehicle is predicted to take in response to the object, a predicted movement of the object, or a combination thereof.

4 . The method of claim 2 , wherein the relevance level of the object is determined based on sensor data indicating that the user is reacting to the object, and wherein the change of the context of the object includes a location change, a speed change, a direction change, a light change, a weather change, a visibility change, or a combination thereof.

5 . The method of claim 4 , wherein the sensor data is collected using at least one sensor configured to detect a gaze, a heartrate, a sweating rate, an eye movement, a body movement, or a combination thereof of the user.

6 . The method of claim 1 , wherein the one or more displays, the one or more speakers, the one or more physical actuators, or a combination thereof include at least one user interface element configured to receive a user input to specify at least one parameter of the simulation, and wherein the simulation is generated based on the at least one parameter, and wherein the at least one user interface element includes at least one knob, at least one joystick, at least one rollerball, at least one trackball-based interface, at least one touch screen, at least one microphone, or a combination thereof.

7 . The method of claim 6 , wherein the at least one parameter includes a time duration, a starting time, a time direction, or a combination thereof.

8 . The method of claim 1 , further comprising:
determining a confidence level of the future event, wherein at least one characteristic of the simulation of the future event is presented based on the confidence level.

9 . The method of claim 1 , wherein the future event includes a plurality of potential outcomes, the method further comprising:
presenting respective representations of at least two of the plurality of potential outcomes in the one or more displays, the one or more speakers, the one or more physical actuators, or a combination thereof, wherein the one or more displays include a vehicle windshield, a heads-up display, a handheld display, an integrated dashboard, a headrest display, a goggle, an eyeglass device, or a combination thereof.

10 . The method of claim 1 , wherein the one or more physical actuators actuate one or more physical sensations predicted in the simulation of the future event.

11 . The method of claim 1 , wherein the one or more displays, the one or more speakers, the one or more physical actuators, or a combination thereof include or communicate with at least one user interface element configured to receive a user input to modify the operation of the vehicle in response to the future event

12 . The method of claim 11 , further comprising:
determining a negative reaction of the user to the simulation of the future event; and modifying the operation of the vehicle to prevent the future event from happening, based on the negative reaction, the user input to modify the operation of the vehicle, or a combination thereof.

13 . An apparatus comprising:
at least one processor; and at least one memory including computer program code for one or more programs, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following,
collect sensor data, contextual data, or a combination thereof during an operation of a vehicle;
process the sensor data, the contextual data, or a combination thereof to predict an event that is expected to occur proximate to the vehicle, wherein the event involves a change of context of at least one object external to the vehicle;
generate a simulation of the event; and
before the future event occurs, performing, by the one or more processors, one or more of:
(i) cause one or more displays to display the simulation of the future event in the vehicle, (ii) cause one or more speakers to output audio in the vehicle in accordance with the simulation of the future event, or (iii) cause one or more physical actuators in the vehicle to actuate in accordance with the simulation of the future event.

14 . The apparatus of claim 13 , wherein the one or more displays, the one or more speakers, the one or more physical actuators, or a combination thereof include or communicate with at least one user interface element configured to receive a user input to specify at least one parameter of the simulation, and wherein the simulation is generated based on the at least one parameter.

15 . The apparatus of claim 13 , wherein the apparatus is further caused to:
determine a confidence level of the future event, wherein at least one characteristic of the simulation of the future event is presented based on the confidence level.

16 . The apparatus of claim 13 , wherein the future event includes a plurality of potential outcomes, the apparatus is further caused to:
present respective representations of at least two of the plurality of potential outcomes in the one or more displays, the one or more speakers, the one or more physical actuators, or a combination thereof.

17 . A non-transitory computer-readable storage medium for generating vulnerable road user data for a geographic database, carrying one or more sequences of one or more instructions which, when executed by one or more processors, cause an apparatus to perform:
determining an object proximate to a vehicle based on a relevance level of the object to a user of the vehicle; predicting a future event involving a change of context of the object based on sensor data, contextual data, or a combination, wherein the object is external to the vehicle; generating a simulation of the future event; and before the future event occurs, performing, by the one or more processors, one or more of: (i) causing one or more displays to display the simulation of the future event in the vehicle, (ii) causing one or more speakers to output audio in the vehicle in accordance with the simulation of the future event, or (iii) causing one or more physical actuator

Assignees

• Here Global Bv

Inventors

• Tuukkanen Marko
• Beaurepaire Jerome

Classifications

• B60W2050/146

  Display means · CPC title

• B60W50/14

  Means for informing the driver, warning the driver or prompting a driver intervention · CPC title

• A61B5/0531

  Measuring skin impedance · CPC title

• A61B5/024

  Measuring pulse rate or heart rate · CPC title

• A61B5/7445

  Display arrangements, e.g. multiple display units · CPC title