Methods and systems for providing remote assistance based on a tactile event

Patent metadata

Field	Value
Publication number	US-10466694-B1
Application number	US-201715704231-A
Country	US
Kind code	B1
Filing date	Sep 14, 2017
Priority date	Jul 11, 2017
Publication date	Nov 5, 2019
Grant date	Nov 5, 2019

Abstract

Examples described may enable provision of remote assistance for an autonomous vehicle. An example method includes a computing system operating in a rewind mode. In the rewind mode, the system may be configured to provide information to a remote assistance operated based on a remote-assistance triggering criteria being met, such as the autonomous vehicle experiencing a tactile event. When the triggering criteria is met, the remote assistance system may provide data from the time leading up to when the remote-assistance triggering criteria was met that was capture of the environment of autonomous vehicle to the remote assistance operator. Based on viewing the data, the remote assistance operator may provide and input to the system that causes a command to be issued to the autonomous vehicle.

First claim

What is claimed is:

1. A method of providing remote assistance for an autonomous vehicle by a remote assistance system, the method comprising:
receiving image data from the autonomous vehicle of an environment of the autonomous vehicle, wherein the received image data is stored in a memory of the remote assistance system;
receiving an indication of a tactile event, the tactile event representing a movement of the autonomous vehicle from the autonomous vehicle;
in response to receiving the indication of the tactile event, providing at least one image corresponding to a predetermined period of time before the tactile was detected to an operator from the memory, wherein at least one image comprises image data related to the tactile event;
receiving an operator input; and
in response to the operator input, providing an instruction via a network to the autonomous vehicle for execution by the autonomous vehicle.

2. The method of claim 1 , wherein the at least one image comprises a video.

3. The method of claim 2 , wherein the video is recorded by a camera of the autonomous vehicle.

4. The method of claim 3 , wherein the video comprises a predetermined length of video stored from the autonomous vehicle.

5. The method of claim 3 , further comprising receiving an operator request for additional image data, and responsively providing additional image data from the memory.

6. The method of claim 1 , wherein the instruction to the autonomous vehicle comprises an instruction for the autonomous vehicle to alter a current driving instruction.

7. The method of claim 1 , wherein the operator input comprises an indication of a cause of the tactile event.

8. The method of claim 1 , wherein the indication of the tactile event comprises force information relating to the tactile event.

9. The method of claim 8 , further comprising providing an operator with a simulation of the tactile event, wherein the simulation enables the operator to feel the tactile event.

10. A system comprising:
a communication unit configured to communicate with an autonomous vehicle via a network, wherein the communication unit is configured to:
receive data from the autonomous vehicle comprising image data and an indication of a tactile event representing a movement of the autonomous vehicle,
send instructions to the autonomous vehicle for execution by the autonomous vehicle; and

a memory configured to store the received data;
a processor configured to:
in response to receiving the indication of a tactile event, provide at least one image corresponding to a predetermined period of time before the tactile was detected to an operator from the memory, wherein at least one image comprises image data related to the tactile event,
receive an operator input,
determine at least one instruction for the autonomous vehicle, and
cause the communication unit to send the at least one instruction to the autonomous vehicle.

11. The system of claim 10 , wherein the image data comprises video data.

12. The system of claim 11 , wherein the video comprises a predetermined length of video stored from the autonomous vehicle.

13. The system of claim 11 , wherein the processor is further configured to receive an operator request for additional image data and responsively provide additional image data from the memory.

14. The system of claim 10 , wherein the at least one instruction for the autonomous vehicle comprises an instruction for the autonomous vehicle to alter a current driving instruction.

15. The system of claim 10 , further comprising the processor causing an operator to feel a simulation of the tactile event.

16. An article of manufacture including a non-transitory computer-readable medium having stored thereon instructions that, when executed by a processor cause the processor to perform operations comprising:
receiving image data from the autonomous vehicle of an environment of the autonomous vehicle, wherein the received image data is stored in a memory of the remote assistance system;
receiving an indication of a tactile event representing a movement of the autonomous vehicle from the autonomous vehicle;
in response to receiving an indication of the tactile event, providing at least one image corresponding to a predetermined period of time before the tactile was detected to an operator from the memory, wherein at least one image comprises image data related to the tactile event;
receiving an operator input; and
in response to the operator input, providing an instruction via a network to the autonomous vehicle for execution by the autonomous vehicle.

17. The article of manufacture of claim 16 , wherein the at least one image comprises a video.

18. The article of manufacture of claim 17 , wherein the video comprises a predetermined length of video stored from the autonomous vehicle.

19. The article of manufacture of claim 17 , further comprising receiving an operator request for additional image data, and responsively providing additional image data from the memory.

20. The article of manufacture of claim 16 , further comprising providing an operator with a simulation of the tactile event, wherein the simulation enables the operator to feel the tactile event.

Assignees

• Waymo Llc

Inventors

• Fairfield Nathaniel
• Herbach Joshua

Classifications

• G05D1/0016

  characterised by the operator's input device (input arrangements for computing systems in general G06F3/00) · CPC title

• G05D1/0038 Primary

  by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation (images analyzed by a computer and used for automatic navigation G05D1/0246) · CPC title

• G05D1/005

  by providing the operator with signals other than visual, e.g. acoustic, haptic · CPC title

• G05D1/0044

  by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps (maps used for automatic navigation G05D1/0274; flight directors G01C23/005) · CPC title

• G05D1/0027 Primary

  involving a plurality of vehicles, e.g. fleet or convoy travelling (fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00) · CPC title