Exception situation playback for tele-operators

Patent metadata

Field	Value
Publication number	US-11215987-B2
Application number	US-201916427742-A
Country	US
Kind code	B2
Filing date	May 31, 2019
Priority date	May 31, 2019
Publication date	Jan 4, 2022
Grant date	Jan 4, 2022

Abstract

Resolving an exception situation in autonomous driving includes receiving an assistance request to resolve the exception situation from an autonomous vehicle (AV); identifying a solution to the exception situation; forwarding the solution to a tele-operator; receiving a request for playback data from the tele-operator; receiving, from the AV, the playback data; and obtaining, from the tele-operator, a validated solution based on the tele-operator using the playback data. The playback data includes snapshots ni of data related to autonomous driving stored at the AV at respective consecutive times ti, for i=1, . . . , N.

First claim

What is claimed is:

1. A method for resolving an exception situation in autonomous driving, comprising:
receiving an assistance request to resolve the exception situation from an autonomous vehicle (AV);
identifying a solution to the exception situation;
forwarding the solution to a tele-operator;
receiving a request for playback data from the tele-operator;
receiving, from the AV, the playback data, wherein the playback data comprises snapshots n i of data related to autonomous driving stored at the AV at respective consecutive times t i , wherein i=1, . . . , N;
displaying, to the tele-operator, a time-warp of the playback data such that a first portion of the playback data including activity related to the exception situation is played at a first speed and a second portion of the playback data not including any activity related to the exception situation is played at a second speed that is faster than the first speed;
highlighting, in a playback of the playback data and for display to the tele-operator, only real world objects causing the exception situation by:
starting from a last snapshot in the playback data, identifying an object that caused the exception situation;
identifying the object in every previous snapshot of the snapshots; and
highlighting the object during the playback;

obtaining, from the tele-operator, a validated solution based on the tele-operator using the playback data; and
autonomously operating the AV according to the validated solution.

2. The method of claim 1 , further comprising:
prior to obtaining the validated solution from the tele-operator, sequentially displaying, to the tele-operator, images from a front facing camera of the AV.

3. The method of claim 1 , further comprising:
receiving, from the tele-operator, a perspective-shift command including a perspective; and
sequentially displaying, to the tele-operator, images from the perspective.

4. The method of claim 1 , further comprising:
receiving, from the tele-operator, an annotation of the playback data, the annotation describing the exception situation and a performance of the AV with respect to the exception situation.

5. The method of claim 1 , further comprising:
transmitting the validated solution to the AV.

6. A method for recording exception situations by an autonomous vehicle (AV), comprising:
storing, in a memory, at consecutive times t i respective snapshots n i of data related to autonomous driving, wherein i=1, . . . , N;
identifying an exception situation occurring at time t n+1 ;
in response to determining that an assistance of a tele-operator is not required, replacing a first snapshot n 1 with a new snapshot corresponding to the time t n+1 ; and
in response to determining that the assistance of the tele-operator is required:
freezing the snapshots n i , for i=1, . . . , N;
transmitting playback data comprising the snapshots n i to an autonomous system;
notifying, via a notification, an agent proximal to the AV of the exception situation, wherein the notification comprises an identifier of the AV and an identification of the exception situation;
displaying, to the tele-operator, at least a portion of the playback data;
highlighting, while displaying the at least the portion of the playback data, real world objects causing the exception situation, wherein highlighting the real world objects causing the exception situation comprises:
starting from a last snapshot in the playback data, identifying a real world object that caused the exception situation;
identifying the real world object in every previous snapshot of the playback data; and
highlighting the real world object while displaying the at least the portion of the playback data;

receiving a validated solution from the tele-operator, wherein the tele-operator uses the snapshots n i to generate the validated solution; and
operating the AV according to the validated solution.

7. The method of claim 6 , wherein transmitting the snapshots n i to the autonomous system comprises:
transmitting the snapshots n i to the autonomous system in response to receiving a request from the autonomous system for the snapshots n i .

8. The method of claim 6 , wherein the memory is a circular buffer.

9. The method of claim 8 ,
wherein the first snapshot n 1 is stored at a location L of the circular buffer, and
wherein replacing the first snapshot n 1 with the new snapshot corresponding to the time t n+1 comprises:
replacing the first snapshot n 1 at the location L with the new snapshot.

10. The method of claim 6 , wherein the data related to autonomous driving comprises raw sensor data.

11. The method of claim 6 , wherein the data related to autonomous driving comprises processed data of raw sensor data.

12. The method of claim 6 , wherein freezing the snapshots n i , for i=1, . . . , N further comprises:
storing the new snapshot corresponding to the time t n+1 in another memory that is different than the memory.

13. A system for resolving an exception situation in autonomous driving, comprising:
a memory; and
a processor, the memory includes instructions executable by the processor to:
receive an assistance request to resolve the exception situation from an autonomous vehicle (AV);
receive a request for playback data from a tele-operator;
receive from the AV the playback data, wherein the playback data comprises snapshots n i of data related to autonomous driving stored at the AV at respective consecutive times wherein i=1, . . . , N;
notify an agent proximal to the AV of the exception situation;
receive, from the agent, agent playback data, wherein the agent freezes the agent playback data in response to being notified of the exception situation;
starting from a last snapshot in the playback data, identify a real world object that caused the exception situation;
identify the real world object in every previous snapshot of the playback data; and
highlight the real world object while displaying the playback data to the tele-operator;
obtain, from the tele-operator, a validated solution based on the tele-operator using the playback data and the agent playback data; and
operating the AV according to the validated solution.

14. The system of claim 13 , wherein the instructions further comprise instructions to:
sequentially display, to the tele-operator, images from a front facing camera of the AV.

15. The system of claim 13 , wherein the instructions further comprise instructions to:
receive, from the tele-operator, a perspective-shift command including a perspective; and
sequentially display, to the tele-operator, images from the perspective.

16. The system of claim 13 , wherein the instructions further comprise instructions to:
receive, from the tele-operator, an annotation of the playback data.

17. The system of claim 13 , wherein the instructions further comprise instructions to:
transmit the validated solution to the AV.

18. The system of claim 13 , wherein the instructions further comprise instructions to:
receive, from the tele-operator, an annotation of the playback data, subsequent to transmitting the validated solution to the AV.

Assignees

• Nissan North America Inc
• Univ Massachusetts
• Renault Sas

Inventors

• Ostafew Christopher
• Vagadia Astha
• Baig Najamuddin
• James Viju
• Witwicki Stefan
• Zilberstein Shlomo

Classifications

• G05D2201/0213

  Physics · mapped topic

• G05D1/0038 Primary

  by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation (images analyzed by a computer and used for automatic navigation G05D1/0246) · CPC title

• G05D1/0061

  for transition from automatic pilot to manual pilot and vice versa · CPC title

• G05D1/0088

  characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00) · CPC title

• G05D1/0027 Primary

  involving a plurality of vehicles, e.g. fleet or convoy travelling (fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00) · CPC title