Predictive teleoperator situational awareness

Patent metadata

Field	Value
Publication number	US-10976732-B2
Application number	US-201916288009-A
Country	US
Kind code	B2
Filing date	Feb 27, 2019
Priority date	Jul 7, 2017
Publication date	Apr 13, 2021
Grant date	Apr 13, 2021

Abstract

A teleoperator device may be configured to obtain a request for teleoperator assistance from a driverless vehicle and obtain teleoperator data in response to the request. The teleoperator device may also be configured to record at least some of the teleoperator input and/or guidance transmitted to the driverless vehicle based on the teleoperator input. Upon receiving a subsequent request, the teleoperator device may be configured to reproduce at least part of the former teleoperator input and/or to provide an option to activate guidance associated with the teleoperator input. The teleoperator device may also be configured to train a model and/or use a model to determine from vehicle data an option for presentation via a teleoperator interface and/or a presentation configuration of the teleoperator interface.

First claim

What is claimed is:

1. A method comprising:
obtaining data from a vehicle;
determining a representation of data based at least in part on a subset of the data, wherein the subset comprises at least one of a request associated with the vehicle, sensor data, a detected object, event data, vehicle state data, or environmental data;
providing, as input to a machine-learned model, the representation of data;
receiving, from the machine-learned model and based at least in part on the representation of data, at least one of a teleoperator option or a presentation configuration;
causing at least one of presentation of the teleoperator option via a user interface or configuration of the user interface based at least in part on the presentation configuration; and
modifying the machine-learned model, wherein modifying the machine-learned model comprises adjusting a parameter of the machine-learned model to decrease a time between receiving the data from the vehicle and transmitting a guidance signal to the vehicle.

2. The method of claim 1 , further comprising:
transmitting the guidance signal to the vehicle based at least in part on a selection received via the user interface; and
receiving an indication that the vehicle received the guidance signal and successfully operated in view of an event that caused the first vehicle to transmit the first request.

3. The method of claim 1 , further comprising:
receiving a teleoperator interaction via the user interface that comprises at least one of a teleoperator input that differs from the teleoperator option, a modification of a presentation at the user interface, the guidance signal, or a result from the vehicle; and
wherein modifying the machine-learned model is based on at least one of the teleoperator input, the modification, the guidance signal, or the result from the vehicle.

4. The method of claim 1 , wherein determining the representation of data comprises populating a tensor or a vector with one or more values associated with at least one of the request, the sensor data, the detected object, the event data, the vehicle state data, or the environmental data, wherein the tensor or the vector is configured as input for a machine-learned model.

5. The method of claim 1 , wherein modifying the machine-learned model further comprises modifying the machine-learned model to decrease a number of user interface interactions required to at least one of modify the presentation to achieve the modification or send the guidance signal.

6. A system comprising:
one or more processors; and
at least one memory having stored thereon processor-executable instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
obtaining data from a vehicle;
determining a representation of data based at least in part on a subset of the data, where the subset comprises at least one of a request, sensor data, a detected object, event data, vehicle state data, or environmental data;
providing, as input to a machine-learned model, the representation of data;
receiving, from the machine-learned model and based at least in part on the representation of data, at least one of a teleoperator option or a presentation configuration;
causing at least one of presentation of the teleoperator option via a user interface or configuration of the user interface based at least in part on the presentation configuration; and
modifying the machine-learned model to decrease a time between receiving the data from the vehicle and transmitting a response to the vehicle.

7. The system of claim 6 , wherein the operations comprise causing presentation of the teleoperator option, and the operations further comprise:
receiving, via a user interface, a selection of the teleoperator option; and
transmitting a guidance signal to the vehicle based on the selection.

8. The system of claim 7 , wherein the selection, via the user interface, of the teleoperator option causes a first action, and wherein the operations further comprise:
receiving a second selection, via the user interface, associated with a second action that is different than the first action;
receiving an indication that the first vehicle received the guidance signal and successfully operated in view of an event that caused the first vehicle to transmit the first request; and
modifying the machine-learned model based at least in part on at least one of a state of the presentation configuration associated with the first action or a difference between the first action and the second action.

9. The system of claim 6 , wherein the operations comprise causing presentation of the teleoperator option, and the teleoperator option comprises one or more options for presentation via a user interface, the options comprising at least one of a first option to accept a proposed guidance, a second option to reject the proposed guidance, or a third option to modify the proposed guidance.

10. The system of claim 6 , wherein the operations further comprise:
transmitting a guidance signal to the vehicle based at least in part on a selection associated with the user interface; and
receiving an indication that the vehicle received the guidance signal and a measure of success of operation of the vehicle in view of an event.

11. The system of claim 10 , wherein:
the machine-learned model includes a neural network trained to receive the representation of data as an input and configured to output at least one of the teleoperator option or the presentation configuration;
the teleoperator option is associated with a first action, and
the operations further comprise modifying one or more parameters of the neural network based at least in part on at least one of the indication, the representation of data, or a change to the presentation configuration, a modification of the first action, or a second action associated with the guidance signal.

12. The system of claim 6 , wherein determining the representation of data comprises populating at least one of a tensor or a vector with one or more values associated with the request, the sensor data, the detected object, the event data, the vehicle state data, or the environmental data, wherein the tensor or the vector is configured as input for the machine-learned model.

13. A non-transitory computer-readable storage medium having processor-executable instructions stored thereupon which, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving data from a vehicle;
determining, based at least in part on the data, a representation of data comprising at least one of a tensor or a vector;
providing, as input to a machine-learned model, the representation of data;
receiving, from the machine-learned model and based at least in part on the representation of data, at least one of a teleoperator option or a presentation configuration; and
causing at least one of presentation of the teleoperator option via a user interface or configuration of the user interface based at least in part on the presentation configuration;
receiving a teleoperator interaction via the user interface;
transmitting a guidance signal to the vehicle based at least in part on the teleoperator interaction; and
modifying the machine-learned model to decrease a time between receiving the data from the vehicle and at least one of receiving the teleoperator interaction response to the vehicle and transmitting the guidance signal.

14. The non-transitory computer-readable storage medium of claim 13 , wherein transmitting the guidance signal is based at least in part on s

Assignees

• Zoox Inc

Inventors

• Lockwood Amanda Lee Kelly
• Gogna Ravi
• Linscott Gary
• Orecchio Paul
• Xie Dan
• Rege Ashutosh Gajanan
• Levinson Jesse Sol

Classifications

• G05D1/2279

  involving allocation of control between two or more remote operators, e.g. tele-assistance · CPC title

• G05D1/6987

  by centralised control off-board any of the vehicles · CPC title

• G05D1/2247

  providing the operator with simple or augmented images from one or more cameras · CPC title

• G05D2107/13

  Spaces reserved for vehicle traffic, e.g. roads, regulated airspace or regulated waters · CPC title

• G05D2105/22

  of humans · CPC title