Remote assistance for an autonomous vehicle in low confidence situations

Patent metadata

Field	Value
Publication number	US-9465388-B1
Application number	US-201414195663-A
Country	US
Kind code	B1
Filing date	Mar 3, 2014
Priority date	Mar 3, 2014
Publication date	Oct 11, 2016
Grant date	Oct 11, 2016

Abstract

Example systems and methods enable an autonomous vehicle to request assistance from a remote operator when the vehicle's confidence in operation is low. One example method includes operating an autonomous vehicle in a first autonomous mode. The method may also include identifying a situation where a level of confidence of an autonomous operation in the first autonomous mode is below a threshold level. The method may further include sending a request for assistance to a remote assistor, the request including sensor data representative of a portion of an environment of the autonomous vehicle. The method may additionally include receiving a response from the remote assistor, the response indicating a second autonomous mode of operation. The method may also include causing the autonomous vehicle to operate in the second autonomous mode of operation in accordance with the response from the remote assistor.

First claim

What is claimed is:

1. A computer-implemented method comprising:
operating an autonomous vehicle in a first autonomous mode;
identifying, by a control system of the autonomous vehicle, a situation where a level of confidence of an autonomous operation in the first autonomous mode is below a threshold level;
sending, by the control system of the autonomous vehicle, a request for assistance to a remote assistor, the request comprising sensor data representative of a portion of an environment of the autonomous vehicle, the request further comprising a plurality of proposed autonomous modes of operation for the autonomous vehicle to proceed in the identified situation;
receiving, by the control system of the autonomous vehicle, a response from the remote assistor, the response indicating a second autonomous mode of operation, wherein the second autonomous mode of operation is selected from the plurality of proposed autonomous modes of operation for the autonomous vehicle to proceed in the identified situation; and
causing the autonomous vehicle to operate in the second autonomous mode of operation in accordance with the response from the remote assistor.

2. The method of claim 1 , wherein identifying the situation where the level of confidence of the autonomous operation in the first autonomous mode is below the threshold level comprises determining when an amount of uncertainty associated with a position of the autonomous vehicle is above a threshold amount.

3. The method of claim 1 , wherein identifying the situation where the level of confidence of the autonomous operation in the first autonomous mode is below the threshold level comprises determining when the autonomous vehicle is unable to move for a certain amount of time.

4. The method of claim 1 , wherein identifying the situation where the level of confidence of the autonomous operation in the first autonomous mode is below the threshold level comprises determining when an amount of uncertainty associated with identification of one or more objects within the environment is above a threshold amount.

5. The method of claim 1 , wherein identifying the situation where the level of confidence of the autonomous operation in the first autonomous mode is below the threshold level comprises receiving a warning signal from one or more sensor systems of the autonomous vehicle.

6. The method of claim 1 , wherein the sensor data comprises:
a sensor data representation of the portion of the environment of the autonomous vehicle, the sensor data representation comprising a representation based on sensor data collected by the vehicle and used by the vehicle to determine the first mode of operation; and
a live video stream of the portion of the environment from one or more cameras on the autonomous vehicle.

7. The method of claim 1 , wherein sending the request for assistance comprises requesting assistance from a passenger of the autonomous vehicle.

8. The method of claim 1 , wherein sending the request for assistance comprises sending one or more timestamps indicative of when the sensor data was collected to the remote assistor.

9. The method of claim 1 , wherein:
operating the autonomous vehicle in the first autonomous mode comprises operating the autonomous vehicle with a first level of caution, wherein the first level of caution indicates an amount of certainty in an autonomous operation required by the vehicle to execute the autonomous operation; and
operating the autonomous vehicle in the second autonomous mode comprises operating the autonomous vehicle with a second level of caution, wherein the second level of caution indicates a different amount of certainty in an autonomous operation required by the vehicle to execute the autonomous operation.

10. The method of claim 1 , wherein:
operating the autonomous vehicle in the first autonomous mode comprises operating the autonomous vehicle within a first range of speeds; and
operating the autonomous vehicle in the second autonomous mode comprises operating the autonomous vehicle within a second range of speeds.

11. A computer-implemented method comprising:
receiving a request for assistance from an autonomous vehicle, the request comprising:
a sensor data representation of an environment of the autonomous vehicle, the sensor data representation comprising a representation based on sensor data collected by the vehicle;
a live video stream of a portion of the environment from one or more cameras on the autonomous vehicle; and
a plurality of proposed autonomous modes of operation for the autonomous vehicle;
providing for display of a graphical interface, wherein the graphical interface comprises:
a first sub-window showing the sensor data representation of the environment;
a second sub-window showing the live video stream of the portion of the environment; and
a list of the plurality of proposed autonomous modes of operation;

receiving a selected proposed mode of autonomous operation for the autonomous vehicle from the list via the graphical interface; and
sending a response to the autonomous vehicle indicating to switch to the selected proposed mode of autonomous operation.

12. The method of claim 11 , wherein the graphical interface is presented to a human operator.

13. The method of claim 11 , wherein the sensor data representation is overlayed on top of the live video stream within the graphical interface.

14. The method of claim 11 , further comprising:
receiving one or more timestamps indicative of when the sensor data was collected;
based on the one or more timestamps, determining a latency amount indicative of an age of the sensor data; and
providing for display of the latency amount within the graphical interface.

15. The method of claim 11 , wherein the response comprises an area of focus, wherein the area of focus comprises a particular portion of the environment for the autonomous vehicle to watch for a change within the environment in order to proceed within the selected proposed mode of autonomous operation.

16. An autonomous vehicle, comprising:
a control system configured to:
operate the autonomous vehicle in a first autonomous mode;
identify a situation where a level of confidence of an autonomous operation in the first autonomous mode is below a threshold level;
send a request for assistance to a remote assistor, the request comprising sensor data representative of a portion of an environment of the autonomous vehicle, the request further comprising a plurality of proposed autonomous modes of operation for the autonomous vehicle to proceed in the identified situation;
receive a response from the remote assistor, the response indicating a second autonomous mode of operation, wherein the second autonomous mode of operation is selected from the plurality of proposed autonomous modes of operation for the autonomous vehicle to proceed in the identified situation; and
cause the autonomous vehicle to operate in the second autonomous mode of operation in accordance with the response from the remote assistor.

17. The vehicle of claim 16 , wherein the control system is configured to identify the situation where the level of confidence of the autonomous operation in the first autonomous mode is below the threshold level by determining when an amount of uncertainty associated with a position of the autonomous vehicle is above a threshold amount.

18. The vehicle of claim 16 , wherein the control system is configured to identify the situation where the level of confidence of the autonomous operation in the first autonomous mo

Assignees

• Google Inc

Inventors

• Fairfield Nathaniel
• Herbach Joshua Seth

Classifications

• B60W2050/0215

  Sensor drifts or sensor failures · CPC title

• H04L67/12

  specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks · CPC title

• G05D1/0044 Primary

  by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps (maps used for automatic navigation G05D1/0274; flight directors G01C23/005) · CPC title

• B60W2556/45

  External transmission of data to or from the vehicle · CPC title

• B60W2556/20

  Data confidence level · CPC title