System to recommend sensor view for quick situational awareness

Patent metadata

Field	Value
Publication number	US-11249479-B2
Application number	US-201916515667-A
Country	US
Kind code	B2
Filing date	Jul 18, 2019
Priority date	Jul 18, 2019
Publication date	Feb 15, 2022
Grant date	Feb 15, 2022

Abstract

A method of exception handing for an autonomous vehicle (AV) includes identifying an exception situation; identifying relevant sensors for the exception situation; identifying relevant tools to the exception situation, the relevant tools usable by a tele-operator to resolve the exception situation; and presenting, on a display of the tele-operator, data from the relevant sensors and the relevant tools.

First claim

What is claimed is:

1. A method of exception handing for an autonomous vehicle (AV), comprising:
identifying an exception situation;
identifying relevant sensors for the exception situation;
identifying relevant tools to the exception situation, the relevant tools usable by a tele-operator to resolve the exception situation,
wherein in a case that the exception situation is identified as a passenger requesting assistance, a first relevant tool is usable by the tele-operator to activate at least one of a speaker or microphone of the AV,
wherein in a case that the exception situation is identified as an obstruction situation, a second relevant tool is a path-drawing tool, and
wherein in a case that a proposed trajectory is received from the AV, a third relevant tool comprises an action usable by the tele-operator to accept the proposed trajectory; and

presenting, on a display of the tele-operator, data from the relevant sensors and the relevant tools.

2. The method of claim 1 ,
wherein the exception situation is identified as an inside-vehicle exception situation, and
wherein the relevant sensors comprise an inside-facing camera of the AV.

3. The method of claim 1 ,
wherein the exception situation is identified as an outside-vehicle exception situation, and
wherein the relevant sensors comprise an outside-facing camera of the AV.

4. The method of claim 1 , further comprising:
receiving an assistance request to resolve the exception situation from the AV.

5. The method of claim 1 , wherein the relevant sensors comprise an off-board sensor.

6. The method of claim 5 , wherein the off-board sensor is an infrastructure sensor.

7. The method of claim 5 , wherein the off-board sensor is an agent sensor.

8. The method of claim 1 , wherein identifying the relevant sensors comprising:
identifying an infrastructure sensor that is within a geofence of the AV.

9. The method of claim 1 , wherein identifying the relevant sensors comprising:
analyzing a first quality of a first image from an on-board sensor;
analyzing a second quality of a second image from an off-board sensor; and
in response to comparing the first quality to the second quality, identifying the off-board sensor to the tele-operator.

10. A system for exception handing for an autonomous vehicle (AV), comprising:
a memory; and
a processor, the processor is configured to execute instructions stored in the memory to:
identify an exception situation;
identify a relevant sensor for the exception situation by steps including:
responsive to first images from an a first sensor being more saturated with light than second images from a second sensor, identify the second sensor as the relevant sensor;

identify relevant tools to the exception situation, the relevant tools usable by a tele-operator to resolve the exception situation; and
present, on a display of the tele-operator, data from the relevant sensor and the relevant tools.

11. The system of claim 10 ,
wherein the exception situation is identified as an inside-vehicle exception situation, and
wherein the relevant sensor is an inside-facing camera of the AV.

12. The system of claim 10 ,
wherein the exception situation is identified as an outside-vehicle exception situation, and
wherein the relevant sensor is an outside-facing camera of the AV.

13. The system of claim 10 , wherein the instructions further comprise instructions to:
receive an assistance request to resolve the exception situation from the AV.

14. The system of claim 10 , wherein the second sensor is an off-board sensor.

15. The system of claim 14 , wherein the off-board sensor is an infrastructure sensor.

16. The system of claim 14 , wherein the off-board sensor is an agent sensor.

17. The system of claim 10 , wherein to identify the relevant sensor comprises to:
identify an infrastructure sensor that is within a geofence of the AV.

18. The system of claim 10 , wherein to identify the relevant sensor further comprises to:
analyze first contents of a first image from an on-board sensor;
analyze second contents of a second image from an off-board sensor; and
in response to comparing the first contents to the second contents, identify the off-board sensor to the tele-operator.

19. A system for exception handing in autonomous driving, comprising:
a memory; and
a processor, the processor is configured to execute instructions stored in the memory to:
receive an assistance request to resolve an exception situation from an autonomous vehicle (AV);
identify an on-board sensor of the AV;
identify an off-board sensor;
select one of the on-board sensor or the off-board sensor as a selected sensor;
present data from the selected sensor to a tele-operator;
present tools to the tele-operator,
wherein the tools comprise a path-drawing tool, a first action usable by the tele-operator to accept a proposed trajectory received from the AV, and a second action usable by the tele-operator to reject the proposed trajectory;

receive, from the tele-operator, a validated solution to the exception situation; and
transmit the validated solution to the AV.

20. The system of claim 19 , wherein to select one of the on-board sensor or the off-board sensor comprises to:
in response to determining that an image from the on-board sensor is saturated with light, select the off-board sensor.

Assignees

• Nissan North America Inc
• Nasa
• Renault Sas

Inventors

• Thakur Siddharth
• Guerin Armelle
• Gyorfi Julius S
• Poulet Kevin
• Laurent Aude
• Allan Mark
• Bentahar Omar

Classifications

• G07C5/008

  communicating information to a remotely located station (transmission systems for measured values G08C) · CPC title

• G07C5/08

  Registering or indicating performance data other than driving, working, idle, or waiting time, with or without registering driving, working, idle or waiting time · CPC title

• G07C5/0808 Primary

  Diagnosing performance data (testing of vehicles G01M17/00; testing of electrical installation on vehicles G01R31/005) · CPC title

• G05D1/0088 Primary

  characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00) · CPC title

• G05D1/0246

  using a video camera in combination with image processing means · CPC title