Teleoperation system and method for trajectory modification of autonomous vehicles

Patent metadata

Field	Value
Publication number	US-9507346-B1
Application number	US-201514932966-A
Country	US
Kind code	B1
Filing date	Nov 4, 2015
Priority date	Nov 4, 2015
Publication date	Nov 29, 2016
Grant date	Nov 29, 2016

Abstract

Various embodiments relate generally to autonomous vehicles and associated mechanical, electrical and electronic hardware, computer software and systems, and wired and wireless network communications to provide an autonomous vehicle fleet as a service. More specifically, systems, devices, and methods are configured to initiate modification of trajectories to influence navigation of autonomous vehicles. In particular, a method may include receiving a teleoperation message via a communication link from an autonomous vehicle, detecting data from the teleoperation message specifying an event associated with the autonomous vehicle, identifying one or more courses of action to perform responsive to detecting the data specifying the event, and generating visualization data to present information associated with the event to a display of a teleoperator computing device.

First claim

What is claimed is:

1. A method comprising:
receiving a teleoperation message via a communication link from an autonomous driverless vehicle;
detecting data from the teleoperation message specifying an event associated with the autonomous driverless vehicle;
identifying one or more courses of action to perform responsive to detecting the data specifying the event;
calculating corresponding ranks for the one or more courses of action; simulating the one or more courses of action to produce one or more simulated courses of action; calculating corresponding simulation values for the one or more simulated courses of action;
generating visualization data to present information associated with the event and at least a subset of the one or more courses of action in association with the corresponding ranks and simulation values to a display of a teleoperator computing device;
receiving data representing selection of a selected course of action from among the one or more courses of action presented on the display; and
transmitting the selected course of action to the autonomous driverless vehicle.

2. The method of claim 1 , further comprising presenting the visualization data on the display to enable a human teleoperator to determine whether to intervene in operation of a planner of the autonomous driverless vehicle.

3. The method of claim 2 , wherein presenting the visualization data comprises depicting a physical environment in which the autonomous driverless vehicle is traveling.

4. The method of claim 2 , wherein presenting the visualization data comprises depicting the subset of the one or more courses of action in association with the corresponding ranks in a portion of the display configured to accept a user input representing a course of action from the human teleoperator.

5. The method of claim 1 , further comprising determining failure to identify a teleoperation message from the autonomous driverless vehicle during an interval of time.

6. The method of claim 5 , wherein determining failure to identify the message comprises detecting absence of a heartbeat-signal as part of the teleoperation message from the autonomous driverless vehicle, where the absence of the heartbeat-signal indicates a loss of communications via the communication link.

7. The method of claim 6 , further comprising receiving a data signal from the teleoperator computing device configured to establish an alternate communication link with the autonomous driverless vehicle.

8. The method of claim 1 , wherein generating the visualization data comprises adjusting content for presentation based on a bandwidth of the communication link.

9. A method comprising:
receiving, at a computing system, sensor data generated from multiple sensors on an autonomous driverless vehicle;
detecting, from the sensor data, degradation of at least one sensor on the autonomous driverless vehicle; simulating one or more actions to produce one or more simulated actions; calculating corresponding simulation values for the one or more simulated actions;
generating, for presentation on a display to a human teleoperator, visualization data pertaining to the sensor data received from the multiple sensors and an indication that the at least one sensor is degraded and the calculated simulation values;
receiving, at the computing system, input from the human teleoperator indicating an action to take to attempt correction of the at least one sensor; and
transmitting the action to the autonomous driverless vehicle.

10. The method of claim 9 , wherein the action comprises an instruction to reduce effect of the degradation.

11. The method of claim 9 , wherein the visualization data to present a visual representation of the sensor degradation.

12. A method comprising:
receiving, at a computing system from an autonomous driverless vehicle, sensor data indicative of an event associated with the autonomous driverless vehicle;
identifying one or more courses of action to perform responsive to the event;
simulating the one or more courses of action to produce one or more simulated courses of action;
calculating corresponding simulation values for the one or more simulated courses of action; and
generating visualization data to present, on a display, pertaining to the one or more courses of action in association with the simulation values; receiving data representing selection of a selected course of action from among the one or more courses of action presented on the display, the selection being based in part on the simulation values corresponding to the one or more courses of action; and transmitting the selected course of action to the autonomous driverless vehicle.

13. The method of claim 12 , further comprising:
calculating corresponding ranks for the one or more courses of action; and
generating visualization data to present at least a subset of the one or more courses of action in association with the corresponding ranks.

14. The method of claim 13 , further comprising:
receiving data representing selection of a selected course of action from among the one or more courses of action presented on the display, the selection being based in part on a combination of the ranks and the simulation values corresponding to the one or more courses of action; and
transmitting the selected course of action to the autonomous driverless vehicle.

15. The method of claim 12 , further comprising presenting the visualization data on the display to enable a human teleoperator to select one of the one or more courses of action.

16. A method comprising:
presenting, on a display of a computing device, a first area containing first visualization data pertaining to an environment within which an autonomous driverless vehicle is operating;
presenting, on the display of the computing device, a second area containing second visualization data pertaining to a listing of actions that could be performed by the autonomous driverless vehicle while operating in the environment, the second visualization data further pertaining to a simulation value of a corresponding action in the listing of actions indicating a result of simulating the corresponding action in a simulation computer; receiving, at the computing device, user selection of a selected action to be performed by the autonomous driverless vehicle; and transmitting the selected action to the autonomous driverless vehicle.

17. The method of claim 16 , wherein the first and second areas are presented simultaneously on the display and the second area is smaller than the first area.

18. The method of claim 16 , further comprising additionally presenting visualization data pertaining to a rank of the corresponding action in the listing of actions in the second area.

Assignees

• Zoox Inc

Inventors

• Levinson Jesse Sol
• Kentley Timothy David
• Sibley Gabriel Thurston
• Gamara Rachad Youssef
• Rege Ashutosh Gajanan
• Linscott Gary

Classifications

• B60W2556/50

  of positioning data, e.g. GPS [Global Positioning System] data · CPC title

• B60W2554/00

  Input parameters relating to objects · CPC title

• G01S2013/9322

  using additional data, e.g. driver condition, road state or weather data · CPC title

• G01S2013/9316

  combined with communication equipment with other vehicles or with base stations · CPC title

• G06F3/1454 Primary

  involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay · CPC title