Method and system for remote assistance of an autonomous agent

Patent metadata

Field	Value
Publication number	US-11565717-B2
Application number	US-202217826655-A
Country	US
Kind code	B2
Filing date	May 27, 2022
Priority date	Jun 2, 2021
Publication date	Jan 31, 2023
Grant date	Jan 31, 2023

Abstract

A system 100 for remote assistance of an autonomous agent can include and/or interface with any or all of: a sensor suite 110, a computing system 120, a communication interface 130, and/or any other suitable components. The system can further optionally include a set of infrastructure devices 140, a teleoperator platform 150, and/or any other suitable components. The system 100 functions to enable information to be exchanged between an autonomous agent and a tele-assist. Additionally or alternatively, the system 100 can function to operate the autonomous agent (e.g., based on remote inputs received from a teleoperator, indirectly, etc.) and/or can perform any other suitable functions.

First claim

We claim:

1. A method comprising:
autonomously controlling a vehicle with an onboard computing system of the vehicle, comprising, at each election step of an election cycle:
determining a vehicle state estimate based on a set of sensor inputs;
determining a set of electable policies based on the vehicle state estimate;
selecting a policy from the set of electable policies;
determining vehicle instructions based on the policy; and
controlling the vehicle according to the vehicle instructions; and

while autonomously controlling the vehicle:
based on the vehicle state estimate, determining satisfaction of an event trigger, wherein the event trigger comprises a probability threshold of an impasse occurrence along a target vehicle path;
based on the event trigger satisfaction, providing a set of outputs to a tele-assist platform; and
updating the set of electable policies to include a tele-operator-approved policy determined with the tele-assist platform.

2. The method of claim 1 , further comprising performing a set of multiple simulations, wherein an individual policy is autonomously selected from the set of electable policies based on a set of outcomes resulting from the set of multiple simulations.

3. The method of claim 2 , further comprising estimating a set of state estimates for a set of dynamic objects in an environment of the vehicle based on the set of sensor inputs, wherein each of the set of multiple simulations is performed based on the set of state estimates and the vehicle state estimate.

4. The method of claim 1 , wherein the vehicle state estimate comprises a vehicle position estimate, and wherein the method further comprises determining a scenario, wherein the set of electable policies are determined by:
determining a set of policies from a predetermined set based on the vehicle position estimate; and
refining the set of policies based on the scenario.

5. The method of claim 1 , further comprising: determining an autonomously unelectable policy with the onboard computing system, wherein the set of outputs provided to the tele-assist platform comprises the autonomously unelectable policy.

6. The method of claim 5 , further comprising: after providing the set of outputs to the tele-assist platform, receiving tele-operator approval of the autonomously unelectable policy from the tele-assist platform and updating the set of electable policies in response.

7. The method of claim 1 , wherein the policy is selected based on a target vehicle path, wherein the method further comprises updating the target vehicle path based on a tele-operator-approved path received from the tele-assist platform.

8. The method of claim 7 , wherein the tele-operator-approved path comprises a set of waypoints which circumvent an obstacle.

9. The method of claim 8 , wherein the set of outputs provided to the tele-assist platform comprises a set of estimated object parameters associated with the obstacle.

10. The method of claim 8 , wherein after updating the target vehicle path based on a tele-operator-approved path, autonomously controlling the vehicle comprises: selecting an updated policy from the set of electable policies, wherein the updated policy deviates from the tele-operator-approved path.

11. The method of claim 10 , wherein the updated policy is associated with a return to the target vehicle path.

12. The method of claim 1 , wherein the event trigger further corresponds to a time threshold.

13. The method of claim 1 , wherein the tele-operator approved policy is received via a wireless connection between the onboard computing system and the tele-assist platform with a latency greater than 50 milliseconds.

14. A method for tele-operation of an autonomous agent, comprising:
at the autonomous agent, determining a first set of inputs;
based on the first set of inputs, determining satisfaction of an event trigger associated with an impasse along a target path;
based on the event trigger satisfaction, providing a set of outputs to a tele-assist platform, the set of outputs comprising a proposed policy which deviates from the target path;
receiving a second set of inputs from the tele-assist platform, the second set of inputs comprising a tele-operator approval of the proposed policy;
based on the first and second sets of inputs, determining a set of electable policies comprising the proposed policy based on the first and second sets of inputs;
at the autonomous agent, selecting a policy from the set of electable policies based on the target path; and
autonomously controlling the vehicle based on the policy.

15. The method of claim 14 , further comprising: prior to selecting the policy, updating the target path based on the second set of inputs.

16. The method of claim 15 , wherein the second set of inputs comprise a set of waypoints.

17. The method of claim 14 , wherein the selected policy is different from the proposed policy.

18. The method of claim 15 , wherein the first set of inputs comprises a position estimate and a scenario, wherein the set of electable policies are determined by:
determining a set of policies from a labeled map based on the position estimate;
expanding the set of policies based on the tele-operator approval; and
refining the set of policies to yield the set of electable policies based on the scenario.

19. The method of claim 14 , wherein the second set of inputs comprises a binary decision by a tele-operator.

20. The method of claim 14 , wherein the policy is selected using a multi-policy decision module of the autonomous agent which defines a tele-operator policy decision node and a set of autonomous policy decision nodes downstream of the tele-operator decision node.

21. A method comprising:
autonomously controlling a vehicle with an onboard computing system of the vehicle, comprising, at each election step of an election cycle:
determining a vehicle state estimate based on a set of sensor inputs;
determining a set of electable policies based on the vehicle state estimate;
selecting a policy from the set of electable policies;
determining vehicle instructions based on the policy; and
controlling the vehicle according to the vehicle instructions; and

while autonomously controlling the vehicle:
based on the vehicle state estimate, determining satisfaction of an event trigger;
based on the event trigger satisfaction, providing a set of outputs to a tele-assist platform; and
updating the set of electable policies to include a tele-operator-approved policy determined with the tele-assist platform,

wherein the tele-operator-approved policy is received via a wireless connection between the onboard computing system and the tele-assist platform with a latency greater than 50 milliseconds.

Assignees

• May Mobility Inc

Inventors

• Kothbauer Melinda
• Sterniak Jeff
• Olson Edwin B

Classifications

• B60W2554/4041

  Position · CPC title

• G05D1/0011

  associated with a remote control arrangement · CPC title

• B60W2554/20

  Static objects · CPC title

• B60W2556/45

  External transmission of data to or from the vehicle · CPC title

• B60W60/0011 Primary

  involving control alternatives for a single driving scenario, e.g. planning several paths to avoid obstacles · CPC title