Remote assist feedback system for autonomous vehicles

Patent metadata

Field	Value
Publication number	US-10501014-B2
Application number	US-201815918043-A
Country	US
Kind code	B2
Filing date	Mar 12, 2018
Priority date	Mar 2, 2018
Publication date	Dec 10, 2019
Grant date	Dec 10, 2019

Abstract

An interaction feedback system for passengers of an autonomous vehicle (AV) can detect a remote assist trigger, corresponding to a remote assist event, from a control system of the AV. The feedback system can then initiate a feedback mode using one or more feedback devices to enable interaction between one or more passengers of the AV and a remote operator based on detecting the remote assist trigger from the AV control system.

First claim

What is claimed is:

1. A passenger feedback system for an autonomous vehicle (AV), comprising:
one or more feedback devices to output remote assist feedback for one or more passengers of the AV; and
a feedback controller comprising one or more processing resources executing instructions that cause the feedback controller to:
detect a remote assist trigger, corresponding to a remote assist event, from a control system of the AV; and
based on detecting the remote assist trigger, transition from a normal operational mode to a feedback mode using the one or more feedback devices to enable two-way interaction between the one or more passengers and a remote operator, wherein the remote operator can transmit control commands to the AV while in feedback mode and the two-way interaction includes voice communication; and

a control interface connecting the feedback controller to the control system of the AV.

2. The passenger feedback system of claim 1 , wherein the executed instructions further cause the feedback controller to:
based on the remote assist trigger, establish a connection, using the communication interface, with a transport service application executing on one or more computing devices of the one or more passengers of the AV to enable the two-way interaction between the one or more passengers and the remote operator.

3. The passenger feedback system of claim 1 , further comprising a communication interface enabling the two-way interaction between the one or more passengers of the AV and the remote operator, wherein the communication interface comprises an on-board display screen displaying a user interface enabling the one or more passengers to communicate with the remote operator.

4. The passenger feedback system of claim 3 , wherein the communication interface further comprises a voice input device enabling the one or more passengers to provide voice input to the remote operator.

5. The passenger feedback system of claim 1 , wherein the remote operator comprises a human operator.

6. The passenger feedback system claim 1 , wherein the remote operator comprises a computer-based, virtual operator.

7. The passenger feedback system of claim 1 , wherein the one or more feedback devices comprise a plurality of output devices including at least one audio output device and at least one visual output device.

8. The passenger feedback system of claim 1 , wherein the remote assist event corresponds to at least one of a perception fault, a connectivity fault, a mechanical fault, or a motion planning fault.

9. The passenger feedback system of claim 1 , wherein the executed instructions further cause the feedback controller to:
prior to initiating the feedback mode, initiate a transitional mode between the normal operational mode and the feedback mode, the transitional mode providing the one or more passengers an alert, using the one or more feedback devices, indicating that the AV has experienced the remote assist event.

10. The passenger feedback system of claim 1 , wherein the remote assist event occurs at the autonomous vehicle.

11. The passenger feedback system of claim 1 , wherein the feedback controller detects a remote assist trigger while in the normal operational mode of the autonomous vehicle.

12. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of an autonomous vehicle (AV), cause the one or more processors to:
detect a remote assist trigger, corresponding to a remote assist event, from a control system of the AV; and
based on detecting the remote assist trigger, transition from a normal operational mode to a feedback mode using one or more feedback devices to enable two-way interaction between one or more passengers of the AV and a remote operator, wherein the remote operator can transmit control commands to the AV while in feedback mode and the two-way interaction includes voice communication.

13. The non-transitory computer readable medium of claim 12 , wherein the executed instructions further cause the one or more processors to:
based on the remote assist trigger, establish a connection with a transport service application executing on one or more computing devices of the one or more passengers of the AV to enable the two-way interaction between the one or more passengers and the remote operator.

14. The non-transitory computer readable medium of claim 12 , wherein the executed instructions cause the one or more processors to enable the two-way interaction between the one or more passengers and the remote operator via a communication interface, wherein the communication interface comprises an on-board display screen displaying a user interface enabling the one or more passengers to communicate with the remote operator.

15. The non-transitory computer readable medium of claim 14 , wherein the communication interface further comprises a voice input device enabling the one or more passengers to provide voice input to the remote operator.

16. The non-transitory computer readable medium of claim 12 , wherein the remote operator comprises a human operator.

17. The non-transitory computer readable medium of claim 12 , wherein the remote operator comprises a computer-based, virtual operator.

18. The non-transitory computer readable medium of claim 12 , wherein the one or more feedback devices comprise a plurality of output devices including at least one audio output device and at least one visual output device.

19. The non-transitory computer readable medium of claim 12 , wherein the remote assist event corresponds to at least one of a perception fault, a connectivity fault, a mechanical fault, or a motion planning fault.

20. A method of providing feedback to passengers of an autonomous vehicle (AV), then method being performed by one or more processors of the AV and comprising:
detecting a remote assist trigger, corresponding to a remote assist event, from a control system of the AV; and
based on detecting the remote assist trigger, transitioning from a normal operational mode to a feedback mode using one or more feedback devices to enable two-way interaction between one or more passengers of the AV and a remote operator, wherein the remote operator can transmit control commands to the AV while in feedback mode and the two-way interaction includes voice communication.

Assignees

• Uatc Llc

Inventors

• Castro Alex
• Nix Molly
• Poeppel Scott
• Chin Sean
• Hanson Eric
• Zhao dennis

Classifications

• G07C5/008 Primary

  communicating information to a remotely located station (transmission systems for measured values G08C) · CPC title

• H04W4/44

  for communication between vehicles and infrastructures, e.g. vehicle-to-cloud [V2C] or vehicle-to-home [V2H] · CPC title

• G07C5/0808

  Diagnosing performance data (testing of vehicles G01M17/00; testing of electrical installation on vehicles G01R31/005) · CPC title

• B60Q9/00 Primary

  Arrangement or adaptation of signal devices not provided for in one of main groups B60Q1/00 - B60Q7/00 {, e.g. haptic signalling} · CPC title

• G05D1/224

  Output arrangements on the remote controller, e.g. displays, haptics or speakers · CPC title