Real-time visualization of autonomous vehicle behavior in mobile applications

Patent metadata

Field	Value
Publication number	US-11645629-B2
Application number	US-201916569488-A
Country	US
Kind code	B2
Filing date	Sep 12, 2019
Priority date	Sep 12, 2019
Publication date	May 9, 2023
Grant date	May 9, 2023

Abstract

Technologies for providing real-time visualizations of a behavior of an autonomous vehicle (AV) associated with a ride request. In some examples, a method for providing real-time visualizations of a behavior of an AV associated with a ride request can include receiving a user request for a ride from an AV, wherein the user request specifies a pick-up location associated with a user; receiving sensor data from one or more sensors associated with the AV; determining, based on the sensor data, a state and context of the AV while the AV is en route to the pick-up location; and presenting, at a display interface, a map depicting one or more visual indicators of the state and context of the AV, the state and context of the AV including a location of the AV and one or more AV operations.

First claim

What is claimed is:

1. A method comprising:
receiving a user request from a user device for a ride from an autonomous vehicle (AV), wherein the user request specifies a pick-up location associated with a user;
receiving data indicating at least one of a state and a context of the AV while the AV is traveling to the pick-up location;
generating, based on the data indicating the at least one of the state and the context of the AV, one or more visual indicators that indicate the at least one of the state and the context of the AV;
providing a map to the user device depicting the one or more visual indicators of the at least one of the state and the context of the AV, the one or more visual indicators providing a description of at least one of a condition along a route to the pick-up location detected by the AV while traveling to the pick-up location, a response to the condition that the AV is predicted to perform, and a control function that is being performed or has been performed by the AV while traveling to the pick-up location;
determining, based on sensor data from one or more sensors associated with the AV, a change in the at least one of the state and the context of the AV, the change occurring while the AV is traveling to the pick-up location;
providing, to the user device, one or more updated visual indicators identifying the change in the at least one of the state and the context of the AV, wherein the one or more updated visual indicators identify at least one of a status of a turn signal on the AV, a status of a brake light on the AV, a steering wheel rotation measurement of a steering wheel on the AV, and a u-turn operation;
receiving a video feed captured by one or more camera sensors on the AV, the video feed depicting a local scene associated with the AV;
identifying a parking location for the user to enter the AV;
identifying an object in the video feed corresponding to the pick-up location for the user to enter the AV; and
providing, to the user device, the video feed and a message identifying the object as the parking location for the user to enter the AV within the pick-up location, wherein the user device concurrently displays the video feed and the message identifying the object as the parking location for the user to enter the AV.

2. The method of claim 1 , wherein the response to the condition or the control function comprise at least one of reversing, parking, braking, turning, activating a turn signal, activating a brake light, and performing a u-turn,
wherein the map presented at the user device depicts the one or more visual indicators while the AV is traveling to the pick-up location.

3. The method of claim 1 , wherein the at least one of the state and the context of the AV comprises at least one of a temperature in an interior of the AV, an AV battery status, a velocity of the AV, a behavior of the AV, and an environment associated with the AV, wherein the condition comprises at least one of traffic, a road closure, a traffic accident, a hazard, a traffic signal, an event in the environment associated with the AV, and an obstacle along a path of the AV.

4. The method of claim 3 , wherein the one or more visual indicators identify at least one of the temperature in the interior of the AV, the AV battery status, and the behavior of the AV.

5. The method of claim 3 , wherein the obstacle along the path of the AV comprises at least one of a stopped vehicle, an animal, a pot hole, a person, emergency personnel, emergency equipment, a construction zone, a street fair, and flooding, wherein the event in the environment associated with the AV comprises at least one of a sporting event, a concert, a traffic signal malfunction, an outdoor event, and a nearby conference, wherein the one or more visual indicators provide a visual representation of at least one of the obstacle along the path of the AV and one or more circumstances associated with the event in the environment.

6. The method of claim 3 , wherein the map depicts one or more aspects of the environment associated with the AV, the method further comprising:
presenting, at the user device, an information section providing at least one of trip information, one or more notifications, and a status of one or more AV controls.

7. The method of claim 1 , wherein at least a portion of the data indicating the at least one of the state and the context of the AV corresponds to the sensor data from the one or more sensors, wherein the one or more sensors comprise at least one of a camera sensor, a light detection and ranging (LIDAR) sensor, a radar sensor, a global positioning system (GPS) sensor, an accelerometer, a motion detection sensor, a speedometer, a light detector, an audio sensor, a braking sensor, an ultrasonic sensor, a position sensor, a steering angle sensor, a steering wheel rotation sensor, a turn signal sensor, a parking sensor, a gear position sensor, a brake light sensor, an odometer, a proximity sensor, a structured-light three-dimensional scanner, an inertial measurement unit, and a temperature sensor, and wherein the at least the portion of the data comprises at least one of an inertial measurement, GPS data, image data, a steering wheel rotation measurement, a measured velocity, a LIDAR point cloud, a radar return measurement, a detected turn signal status, a detected parking status, a detected gear position, a detected brake light status, a proximity measurement, structured-light three-dimensional scanner data, and LIDAR data.

8. The method of claim 1 , further comprising:
tracking, based on the sensor data from the one or more sensors associated with the AV, the at least one of the state and the context of the AV while the AV is en route to the pick-up location; and
presenting, on the map, at least one updated visual indicator of at least one of an updated state and an updated context of the AV, the at least one of the updated state and the updated context of the AV comprising at least one of an updated location of the AV, an updated behavior of the AV, and one or more conditions in an environment associated with the AV.

9. The method of claim 1 , wherein the video feed comprises a 360° external view of the AV.

10. The method of claim 1 , wherein the AV provides the video feed directly to the user based on the AV being within a communication range of the user.

11. A system comprising:
one or more processors; and
memory having instructions stored thereon which, when executed by the one or more processors, cause the one or more processors to:
receive a user request from a user device for a ride from an autonomous vehicle (AV), wherein the user request specifies a pick-up location associated with a user;
receive data indicating at least one of a state and a context of the AV while the AV is traveling to the pick-up location;
generate, based on the data indicating the at least one of the state and the context of the AV, one or more visual indicators that indicate the at least one of the state and the context of the AV;
provide a map to the user device depicting the one or more visual indicators of the at least one of the state and the context of the AV, the one or more visual indicators providing a description of at least one of a condition along a route to the pick-up location detected by the AV while traveling to the pick-up location, a response to the condition that the AV is predicted to perform, and a control function that is being performed or has been performed by the AV while traveling to the pick-up location;
determine, based on sensor data from one or more sensors associated with the AV, a change in the at least one of the state and the context of the AV, the change occurring while the AV is t

Assignees

• Gm Cruise Holdings Llc

Inventors

• Leary Jessica
• Staff Jonathon
• Anderson John Logan
• Meyers Max
• Patel Pratik
• Yang Robin
• Ryan Bradley
• Berlin Josh
• Lin Teresa
• Shiao Stanley
• Magnoli Michael
• Wang Chau-Yan
• Fry Christopher

Classifications

• G06Q10/1093 Primary

  Calendar-based scheduling for persons or groups · CPC title

• G01C21/3697

  Output of additional, non-guidance related information, e.g. low fuel level (G01C21/3679 takes precedence) · CPC title

• G06Q10/02

  Reservations, e.g. for tickets, services or events · CPC title

• H04W4/40

  for vehicles, e.g. vehicle-to-pedestrians [V2P] · CPC title

• G07C5/0825

  using optical means · CPC title