Teleoperation system and method for trajectory modification of autonomous vehicles

Patent metadata

Field	Value
Publication number	US-11314249-B2
Application number	US-201916557899-A
Country	US
Kind code	B2
Filing date	Aug 30, 2019
Priority date	Nov 4, 2015
Publication date	Apr 26, 2022
Grant date	Apr 26, 2022

Abstract

Various embodiments relate generally to autonomous vehicles and associated mechanical, electrical and electronic hardware, computer software and systems, and wired and wireless network communications to provide an autonomous vehicle fleet as a service. More specifically, systems, devices, and methods are configured to initiate modification of trajectories to influence navigation of autonomous vehicles. In particular, a method may include receiving a teleoperation message via a communication link from an autonomous vehicle, detecting data from the teleoperation message specifying an event associated with the autonomous vehicle, identifying one or more courses of action to perform responsive to detecting the data specifying the event, and generating visualization data to present information associated with the event to a display of a teleoperator computing device.

First claim

What is claimed:

1. An autonomous vehicle comprising:
one or more sensors;
a communication interface configured to establish a communications link with a remote computing device;
one or more processors: and
memory storing processor-executable instructions that, when executed by the one or more processors, cause the autonomous vehicle to perform operations comprising:
receiving sensor data from the one or more sensors;
determining one or more attributes of the communications link;
generating an abstraction of the sensor data;
transmitting, based at least in part on an attribute of the one or more attributes exceeding a threshold, one or more of the sensor data to a teleoperator system;
transmitting, based at least in part on the attribute being less than or equal to the threshold, the abstraction of the sensor data to the teleoperator system;
receiving, from the teleoperator system, a teleoperator command; and
controlling, based at least in part on the teleoperator command, the autonomous vehicle.

2. The autonomous vehicle of claim 1 , wherein the sensor data comprises a first representation of a plurality of objects in an environment of the autonomous vehicle and the abstraction comprises a second representation of a subset of the plurality of objects in the environment.

3. The autonomous vehicle of claim 1 , wherein the abstraction comprises a bounding box associated with an object represented in the sensor data.

4. The autonomous vehicle of claim 1 , the operations further comprising:
determining, in the sensor data, object data;
comparing the object data to map data; and
determining, based on the comparing, that the object data varies from the map data,
wherein the abstraction is a representation of the object data exclusive of portions of the sensor data other than the object data.

5. The autonomous vehicle of claim 1 , wherein the one or more attributes of the communications link comprises an available bandwidth.

6. The autonomous vehicle of claim 1 , the operations further comprising:
determining, based at least in part on the sensor data, that an event has occurred,
wherein the transmitting the sensor data or the abstraction of the sensor data is further based at least in part on determining the event has occurred.

7. The autonomous vehicle of claim 6 , the operations further comprising:
retrieving historical sensor data corresponding to a time prior to the event; and
transmitting the historical sensor data or a second abstraction of the historical sensor data to the teleoperator system.

8. The autonomous vehicle of claim 6 , the operations further comprising:
determining a position of the autonomous vehicle in an environment,
wherein the transmitting the sensor data or transmitting the abstraction is further based at least in part in response to the position.

9. A method comprising:
receiving sensor data from a sensor on a vehicle;
generating a first abstraction of the sensor data having a first level of abstraction detail and a second abstraction of the sensor data having a second level of abstraction detail;
determining a bandwidth of a communications link between the vehicle and a remote system;
transmitting, based at least in part on the bandwidth of the communications link and to the remote system, one of the sensor data, the first abstraction, or the second abstraction;
receiving, from the remote system, a command; and
controlling, based at least in part on the command, the vehicle.

10. The method of claim 9 , wherein the sensor data comprises a first representation of a plurality of objects in an environment of the vehicle, the first abstraction comprises a second representation of a first subset of the plurality of objects in the environment and the abstraction comprises a third representation of a second subset of the plurality of objects in the environment.

11. The method of claim 9 , wherein the first abstraction or the second abstraction comprises a bounding box associated with an object represented in the sensor data.

12. The method of claim 11 , wherein the first abstraction or the second abstraction further comprises metadata associated with the bounding box.

13. The method of claim 12 , wherein the metadata comprises one or more of: a classification of the object, a degree of certainty associated with the object, or a velocity of the object.

14. The method of claim 9 , further comprising:
determining, based at least in part on the sensor data, object data;
comparing the object data to map data; and
determining, based on the comparing, that the object data varies from the map data,
wherein the transmitting comprises transmitting the abstraction and the abstraction is a representation of the object data exclusive of portions of the sensor data other than the object data.

15. The method of claim 9 , wherein
the transmitting the sensor data, the first abstraction, or the second abstraction comprises: transmitting the sensor data when the available bandwidth meets or exceeds a first threshold bandwidth, transmitting the first abstraction when the available bandwidth meets or exceeds a second threshold bandwidth lower than the first threshold bandwidth, or transmitting the second abstraction when the available bandwidth is below the second threshold bandwidth.

16. The method of claim 9 , the operations further comprising:
determining, based at least in part on the sensor data, that an event has occurred,
wherein the transmitting the sensor data, the first abstraction, or the second abstraction is based at least in part on determining that the event has occurred.

17. The method of claim 16 , further comprising:
retrieving historical sensor data corresponding to a time prior to the event; and
transmitting the historical sensor data or a second abstraction of the historical sensor data to the remote system.

18. The method of claim 9 , further comprising:
determining a position of the vehicle in a driving environment,
wherein the transmitting the sensor data, the first abstraction, or the second abstraction is based at least in part in response to the position corresponding to a predetermined position.

19. A system comprising:
one or more processors; and
memory storing processor-executable instructions executable by the one or more processors to perform acts comprising:
receiving sensor data from a sensor on a vehicle;
generating a first abstraction of the sensor data at a first level of abstraction detail;
generating a second abstraction of the sensor data at a second level of abstraction detail;
determining an available bandwidth of a communications link between the vehicle and a remote system;
transmitting, based at least in part on the available bandwidth, one of the sensor data, the first abstraction, or the second abstraction to the remote system;
receiving, from the remote system, a command; and
controlling, based at least in part on the command, the vehicle.

20. The system of claim 19 , wherein at least one of the first abstraction or the second abstraction comprises a bounding box associated with an object represented in the sensor data.

21. The system of claim 20 , wherein the bounding box further comprises metadata comprising one or more of: a classification of the object, a degree of certainty associated with the object, or a velocity of the object.

22. The system of claim 19 , the operations further comprising:
determining, based at least in part on the sensor data, obje

Assignees

• Zoox Inc

Inventors

• Levinson Jesse Sol
• Rege Ashutosh Gajanan
• Gamara Rachad Youssef
• Sibley Gabriel Thurston
• Kentley-Klay Timothy David
• Linscott Gary

Classifications

• G01S13/865

  Combination of radar systems with lidar systems · CPC title

• B60W30/00

  Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units · CPC title

• G08G1/202

  Dispatching vehicles on the basis of a location, e.g. taxi dispatching · CPC title

• G01S17/87

  Combinations of systems using electromagnetic waves other than radio waves · CPC title

• G01S13/87

  Combinations of radar systems, e.g. primary radar and secondary radar · CPC title