Advanced control system with multiple control paradigms

Patent metadata

Field	Value
Publication number	US-11740624-B2
Application number	US-201816639216-A
Country	US
Kind code	B2
Filing date	Aug 17, 2018
Priority date	Aug 17, 2017
Publication date	Aug 29, 2023
Grant date	Aug 29, 2023

Abstract

A hybrid control system includes a control agent and a control engine. The control engine is configured to install a master plan to the control agent. The master plan includes a plurality of high-level tasks. The control agent is configured to operate according to the master plan to, for each high-level task of the high-level tasks, obtain one or more low-level controls and to perform the one or more low-level controls to realize the high-level task. The control agent is configured to operate according to the master plan to transition between the plurality of high-level tasks thereby causing a seamless transition between operating at least partially autonomously and operating at least partially based on input from the tele-operator, based at least on context for the control agent, to operate at least partially autonomously and at least partially based on input from the tele-operator during execution of the master plan.

First claim

What is claimed is:

1. A hybrid control system comprising:
a control agent; and
a control engine comprising one or more processors, wherein the control engine is configured to:
install a master plan to the control agent, wherein the master plan comprises a plurality of high-level tasks,
wherein the plurality of high-level tasks includes at least one task for which the control agent is operating at least partially autonomously,
wherein the plurality of high-level tasks for the control agent includes at least one task for which the control agent is operating at least partially based on input from a tele-operator,
wherein the control agent is configured to operate according to the master plan to, for each high-level task of the high-level tasks, obtain or generate one or more low-level controls and to perform the one or more low-level controls to realize the high-level task,
wherein, to obtain or generate the one or more low-level controls, the control agent is configured to dynamically adjust, in real-time, the master plan for a high-level task of the one or more high-level tasks from the installed master plan based on an uncertainty of a scene comprising the control agent,
wherein, to dynamically adjust the master plan, the control agent is configured to dynamically adjust, based on the uncertainty of the scene, the high-level task of the one or more high-level tasks from operating at least partially autonomously to operating at least partially based on input from the tele-operator when at least one control of the low-level controls operating at least partially autonomously matches a control obtained based on input from the tele-operator, and
wherein the control agent is configured to operate according to the adjusted master plan to transition between the plurality of high-level tasks thereby causing a seamless transition between operating at least partially autonomously and operating at least partially based on input from the tele-operator, based at least on a context for the control agent, to operate at least partially autonomously and at least partially based on input from the tele-operator during execution of the adjusted master plan.

2. The system of claim 1 ,
wherein at least one low-level control of the one or more low-level controls obtained for a first high-level task of the one or more high-level tasks is received from the tele-operator, and
wherein at least one low-level control of the one or more low-level controls obtained for a second high-level task of the one or more high-level tasks is autonomously determined by the control agent.

3. The system of claim 1 , wherein the context comprises one or more of a geolocation of the control agent, a state of the control agent, communication conditions, or operator state.

4. The system of claim 1 , wherein the control engine is configured to:
generate the master plan using one or more of historical information, information regarding state of the control agent including either a success state or failure state, capability information for the control agent, or a master plan template for a plurality of control agents.

5. The system of claim 1 , wherein the control engine is configured to:
build up a map of operations that maps a location relative to the map to at least one of: 1) controls performed by the tele-operator or 2) autonomously-performed low-level controls.

6. The system of claim 1 , wherein the control engine comprises:
a simulator module configured to perform a series of simulations to determine a set of low-level controls that best achieves a high-level task of the one or more high-level tasks by simulating commands produced by the set of low-level controls and determining whether the simulated commands produced by the set of low-level controls achieves a threshold of success.

7. The system of claim 1 , wherein the control engine is configured to:
incorporate latency information into calculations performed by a remote simulation module such that one or more commands issued by the tele-operator arrive at the agent when they are needed.

8. The system of claim 1 , wherein the control agent is configured to:
learn low-level controls to increase a level of autonomy for a given high-level task of the one or more high-level tasks.

9. The system of claim 1 , wherein the control agent comprises a vehicle, a truck, a general-purpose robot, a service robot, a drone, or a warehouse robot.

10. The system of claim 1 , wherein, to operate according to the adjusted master plan, the control agent is configured to:
transition between one or more remotely initiated high-level tasks of the plurality of high-level tasks and one or more locally initiated high-level tasks of the plurality of high-level tasks.

11. The system of claim 1 , wherein the control agent is configured to:
output the adjusted master plan to an external system for use by another control agent.

12. The system of claim 1 , wherein the control agent is configured to:
determine a low-level control of the one or more low-level controls based on input from the tele-operator.

13. A method for a hybrid control system comprising:
installing, by a control engine comprising one or more processors, a master plan to a control agent, wherein the master plan comprises a plurality of high-level tasks,
wherein the plurality of high-level tasks includes at least one task for which the control agent is operating at least partially autonomously,
wherein the plurality of high-level tasks includes at least one task for which the control agent is operating at least partially based on input from a tele-operator,
operating, by the control agent, according to the master plan to, for each high-level task of the high-level tasks, obtain or generate one or more low-level controls and to perform the one or more low-level controls to realize the high-level task,
wherein operating according to the master plan comprises dynamically adjusting, in real-time, the master plan for a high-level task of the one or more high-level tasks from the installed master plan based on an uncertainty of a scene comprising the control agent,
wherein dynamically adjusting the master plan comprises dynamically adjusting, based on the uncertainty of the scene, the high-level task of the one or more high-level tasks from operating at least partially autonomously to operating at least partially based on input from the tele-operator when at least one control of the low-level controls operating at least partially autonomously matches a control obtained based on input from the tele-operator, and
operating, by the control agent, according to the adjusted master plan to transition between the plurality of high-level tasks thereby causing a seamless transition between operating at least partially autonomously and operating at least partially based on input from the tele-operator, based at least on a context for the control agent, to operate at least partially autonomously and at least partially based on input from the tele-operator during execution of the adjusted master plan.

14. The method of claim 13 ,
wherein at least one low-level control of the one or more low-level controls obtained for a first high-level task of the one or more high-level tasks is received from the tele-operator, and
wherein at least one low-level control of the one or more low-level controls obtained for a second high-level task of the one or more high-level tasks is autonomously determined by the control agent.

15. The method of claim 13 , wherein the context comprises one or more of a geolocation of the control agent, a state of the control agent, com

Assignees

• Stanford Res Inst Int

Inventors

• Ramamurthy Bhaskar
• Samarasekera Supun
• Low Thomas
• Kothari Manish
• Marcotullio John Peter
• Brookshire Jonathan
• Arodiogbu Tobenna
• Ghani Usman

Classifications

• G05D1/0022 Primary

  characterised by the communication link (data switching networks in general H04L12/00) · CPC title

• G05D1/0088

  characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00) · CPC title

• G05D1/101

  specially adapted for aircraft · CPC title

• B60W60/001

  Planning or execution of driving tasks · CPC title

• G05B17/02

  electric · CPC title