System and method for remotely assisting autonomous vehicle operation

Patent metadata

Field	Value
Publication number	US-10459440-B2
Application number	US-201715398562-A
Country	US
Kind code	B2
Filing date	Jan 4, 2017
Priority date	Jan 4, 2016
Publication date	Oct 29, 2019
Grant date	Oct 29, 2019

Abstract

Systems and methods are provided for remotely assisting an autonomous vehicle. The method includes: aggregating sensor data from the autonomous vehicle; identifying an assistance-desired scenario; generating an assistance request based on the sensor data; transmitting the assistance request to a remote assistance interface; and receiving and processing a response to the assistance request. The remote assistance interface includes a remote assistance interface that is used in generating the response to the assistance request.

First claim

We claim:

1. A system comprising:
an autonomous vehicle, the autonomous vehicle comprising:
one or more sensors; and
an onboard computer that:
collects sensor data from, at least, the one or more sensors;
identifies a scenario, from the collected sensor data, requiring remote assistance to the autonomous vehicle; and
generates an assistance request that identifies the scenario and at least some of the collected sensor data;
wherein the onboard computer includes an assistance router that is disposed onboard the autonomous vehicle and is configured to automatically route the assistance request of the autonomous vehicle to one or more assistance providers;

a remote assistance interface that is communicatively coupled to the assistance router of the onboard computer and is configured to:
receive the assistance request from the autonomous vehicle via the assistance router onboard the autonomous vehicle; and
provide an assistance response to the assistance request of autonomous vehicle after processing the scenario and the collected sensor data identified in the assistance request.

2. The system of claim 1 , wherein:
the assistance router processes the assistance request and identifies one or more destinations of the assistance request based on one or more attributes of the assistance request, wherein the assistance router routes the assistance request to the one or more destinations selected from the remote assistance interface and an artificially-intelligent (AI) expert, wherein the AI expert is distinct from the onboard computer and remote from the autonomous vehicle, and wherein the remote assistance interface provides artificial-intelligently calculated responses to the assistance request.

3. The system of claim 1 , wherein the scenario comprises a situation in which sensing for the autonomous vehicle is impaired.

4. The system of claim 1 , wherein the scenario comprises a situation in which decision making for the autonomous vehicle is impaired.

5. The system of claim 1 , wherein the scenario comprises a situation in which a routing or path for traversing the scenario would run contrary to one or more traffic rules or norms.

6. A method for remotely assisting an autonomous vehicle, the method comprising:
at an onboard computer:
aggregating sensor data from one or more sensors of the autonomous vehicle;
identifying an assistance-desired scenario, wherein the assistance-desired scenario refers to a scenario in which the onboard computer is able to improve vehicle operation and/or safety by augmenting onboard intelligence of the autonomous vehicle using remote assistance;
generating an assistance request at least some of the aggregated sensor data;
transmitting the assistance request to a remote assistance interface that is remote from the autonomous vehicle, via an assistance router that is disposed onboard the autonomous vehicle and is configured to automatically route the assistance request of the autonomous vehicle to one or more assistance providers; and
receiving and processing a response to the assistance request from the autonomous vehicle via the assistance router onboard the autonomous vehicle, via the remote assistance interface.

7. The method of claim 6 , wherein, at the onboard computer, the assistance-desired scenario is identified by identifying an assistance desire score that is based on one or more of a current state of the autonomous vehicle, past or predicted states of the autonomous vehicle, and past, current, or predicted environmental states, and wherein the identified assistance desire score is compared to an assistance desire threshold and if the threshold is exceeded by the assistance desire score, a determination of an assistance-desired scenario is made.

8. The method of claim 6 , wherein the aggregated sensor data includes data observed from one or more of an internal environment of the autonomous vehicle, an external environment of the autonomous vehicle, and operations of systems of the autonomous vehicle, and wherein the onboard computer:
identifies one or more attributes of the aggregated sensor data, and
compares the identified attributes to one or more predetermined assistance scenario lists or an assistance-desired database having stored thereon a plurality of previously-identified scenarios for which remote assistance is required, and
when the identified attributes match an item within the one or more predetermined assistance scenario lists or within the assistance-desired database, identifies a scenario of the autonomous vehicle as an assistance-desired scenario.

9. The method of claim 6 , wherein the aggregated sensor data is selectively parsed to generated a set of assistance data to thereby enabling an intelligent response at the remote assistance interface to the assistance request, and the method further comprises identifying a transmission method for the assistance data assistance request, wherein a first subset of data of the assistance data is transmitted via a low-bandwidth connection and a second subset of data of the assistance data is transmitted via a high-bandwidth connection.

10. The method of claim 6 , wherein the remote assistance interface comprises a remote assistance interface, an artificially-intelligent expert, or a combination of the remote assistance interface and artificially-intelligent expert, and the method further comprises identifying which remote assistance interface to route the transmission of the assistance request by selecting among a remote assistance interface and an artificially-intelligent expert interface.

11. The method of claim 6 , wherein the aggregated data is processed to generate assistance data to be provided together with the assistance request, wherein the assistance data comprises video image data captured by one or more image capturing devices of the autonomous vehicle, and wherein the video image data is processed to drop excess video frames or change compression parameters of the video image data to make the assistance data more suitable for transmission to the remote assistance interface.

12. The method of claim 6 , wherein the response to the assistance request includes a blacklisted path or blacklisted lane that prohibits a use of the blacklisted path or blacklisted lane in a current route or a possible router of the autonomous vehicle.

13. The method of claim 6 , wherein the assistance request response includes (i) redrawing traffic boundary lines, (ii) generating autonomous vehicle control data based on the redrawn traffic boundary lines, and (iii) transmitting the redrawn boundary lines and autonomous vehicle control data to the autonomous vehicle.

14. The method of claim 6 , further comprising:
transforming the response to the assistance request into an automatic response that is automatically implemented via the onboard computer of the autonomous vehicle when a future condition arises that is similar to the scenario that triggered the assistance request.

15. A method for remotely assisting an autonomous vehicle, the method comprising:
at a remote assistance interface that is remote from the autonomous vehicle and coupled to an assistance router that is disposed onboard the autonomous vehicle:
receiving an assistance request from an autonomous vehicle, wherein the assistance request includes assistance request data comprising sensor data obtained by, at least, the autonomous vehicle, the assistance request received from the assistance router and is configured to automatically route the assistance request of the autonomous vehicle to one or more assistance providers;
processing the assistance request da

Assignees

• Gm Global Tech Operations Llc

Inventors

• Rust Ian
• Vogt Kyle
• Bier Solomon
• Gross Drew Allyn

Classifications

• G01C21/3461

  Preferred or disfavoured areas, e.g. dangerous zones, toll or emission zones, intersections, manoeuvre types or segments such as motorways, toll roads or ferries · CPC title

• G07C5/008

  communicating information to a remotely located station (transmission systems for measured values G08C) · CPC title

• G01C21/28

  with correlation of data from several navigational instruments · CPC title

• G01C21/3492

  employing speed data or traffic data, e.g. real-time or historical (traffic control systems for road vehicles involving transmission of navigation instructions to the vehicle G08G1/0968) · CPC title

• G01C21/3438

  Rendezvous; Ride sharing · CPC title