Controlling mechanical systems based on natural language input

Patent metadata

Field	Value
Publication number	US-11725924-B2
Application number	US-202117518417-A
Country	US
Kind code	B2
Filing date	Nov 3, 2021
Priority date	Aug 10, 2021
Publication date	Aug 15, 2023
Grant date	Aug 15, 2023

Abstract

A method is provided. The method includes obtaining an enhanced state graph. The enhanced state graph represents a set of objects within an environment and a set of positions of the set of objects. The enhanced state graph includes a set of object nodes, a set of property nodes and a set of goal nodes to represent a set of objectives. The method also includes generating a set of instructions for a set of mechanical systems based on the enhanced state graph. The set of mechanical systems is configured to interact with one or more of the set of objects within the environment. The method further includes operating the set of mechanical systems to achieve the set of objectives based on the set of instructions.

First claim

What is claimed is:

1. A method, comprising:
obtaining an enhanced state graph, wherein:
the enhanced state graph represents a set of objects within an environment and a set of positions of the set of objects within the environment;
the enhanced state graph comprises a set of object nodes to represent the set of objects;
the enhanced state graph comprises a set of property nodes to represent a set of properties of the set of objects; and
the enhanced state graph comprises a set of goal nodes to represent a set of objectives associated with a subset of the set of objects;

generating a set of instructions for a set of mechanical systems based on the enhanced state graph, wherein the set of mechanical systems is configured to interact with one or more of the set of objects within the environment;
operating the set of mechanical systems to achieve the set of objectives based on the set of instructions;
determining a portion of the environment based on the set of goal nodes, wherein the portion of the environment is associated with the set of objectives; and
determining whether the set of instructions cause the set of mechanical systems to operate within the portion of the environment.

2. The method of claim 1 , further comprising:
identifying a set of locations for the subset of the set of objects; and
identifying a subset of the set of mechanical systems based on the set of locations.

3. The method of claim 2 , wherein generating the set of instructions comprises:
generating the set of instructions for the subset of the set of mechanical systems.

4. The method of claim 1 , wherein generating the set of instructions is further based on the set of goal nodes of the enhanced state graph.

5. The method of claim 1 , further comprising:
in response to determining that the set of instructions do not cause the set of mechanical systems to operate within the portion of the environment, generating a second set of instructions for the set of mechanical systems based on the enhanced state graph.

6. The method of claim 1 , wherein the enhanced state graph is generated based on natural language input from a user.

7. The method of claim 1 , further comprising:
obtaining an updated enhanced state graph;
generating a second set of instructions for the set of mechanical systems based on the updated enhanced state graph, wherein the set of mechanical systems is configured to interact with one or more of the set of objects within the environment; and
operating the set of mechanical systems to achieve the set of objectives based on the set of instructions.

8. An apparatus, comprising:
a memory to store a binary code of a controller; and
a processing device, operatively coupled to the memory, to:
obtain an enhanced state graph, wherein:
the enhanced state graph represents a set of objects within an environment and a set of positions of the set of objects within the environment;
the enhanced state graph comprises a set of object nodes to represent the set of objects;
the enhanced state graph comprises a set of property nodes to represent a set of properties of the set of objects; and
the enhanced state graph comprises a set of goal nodes to represent a set of objectives associated with a subset of the set of objects;

generate a set of instructions for a set of mechanical systems based on the enhanced state graph, wherein the set of mechanical systems is configured to interact with one or more of the set of objects within the environment;
operate the set of mechanical systems to achieve the set of objectives based on the set of instructions;
determine a portion of the environment based on the set of goal nodes, wherein the portion of the environment is associated with the set of objectives; and
determine whether the set of instructions cause the set of mechanical systems to operate within the portion of the environment.

9. The apparatus of claim 8 , wherein the processing devices is further to:
identify a set of locations for the subset of the set of objects; and
identify a subset of the set of mechanical systems based on the set of locations.

10. The apparatus of claim 9 , wherein to generate the set of instructions the processing devices is further to:
generate the set of instructions for the subset of the set of mechanical systems.

11. The apparatus of claim 8 , wherein generating the set of instructions is further based on the set of goal nodes of the enhanced state graph.

12. The apparatus of claim 8 , wherein the processing devices is further to:
in response to determining that the set of instructions do not cause the set of mechanical systems to operate within the portion of the environment, generate a second set of instructions for the set of mechanical systems based on the enhanced state graph.

13. The apparatus of claim 8 , wherein the enhanced state graph is generated based on natural language input from a user.

14. The apparatus of claim 8 , wherein the processing devices is further to:
obtain an updated enhanced state graph;
generate a second set of instructions for the set of mechanical systems based on the updated enhanced state graph, wherein the set of mechanical systems is configured to interact with one or more of the set of objects within the environment; and
operate the set of mechanical systems to achieve the set of objectives based on the set of instructions.

15. A non-transitory computer-readable storage medium having instructions stored thereon that, when executed by a processing device, cause the processing device to:
obtain an enhanced state graph, wherein:
the enhanced state graph represents a set of objects within an environment and a set of positions of the set of objects within the environment;
the enhanced state graph comprises a set of object nodes to represent the set of objects;
the enhanced state graph comprises a set of property nodes to represent a set of properties of the set of objects; and
the enhanced state graph comprises a set of goal nodes to represent a set of objectives associated with a subset of the set of objects;

generate a set of instructions for a set of mechanical systems based on the enhanced state graph, wherein the set of mechanical systems is configured to interact with one or more of the set of objects within the environment;
operate the set of mechanical systems to achieve the set of objectives based on the set of instructions;
determine a portion of the environment based on the set of goal nodes, wherein the portion of the environment is associated with the set of objectives; and
determine whether the set of instructions cause the set of mechanical systems to operate within the portion of the environment.

16. The non-transitory computer-readable storage medium of claim 15 , wherein the instructions further cause the processing device to:
identify a set of locations for the subset of the set of objects; and
identify a subset of the set of mechanical systems based on the set of locations.

17. The non-transitory computer-readable storage medium of claim 16 , wherein generating the set of instructions comprises:
generating the set of instructions for the subset of the set of mechanical systems.

18. The non-transitory computer-readable storage medium of claim 15 , wherein generating the set of instructions is further based on the set of goal nodes of the enhanced state graph.

Assignees

• Palo Alto Res Ct Inc

Inventors

• Mohan Shiwali
• Klenk Matthew
• Shreve Matthew
• Ang Aaron
• Maxwell Iii John Turner
• Evans Kent

Classifications

• G01B7/003 Primary

  for measuring position, not involving coordinate determination (coordinate measuring G01B7/004) · CPC title

• B25J9/163

  learning, adaptive, model based, rule based expert control · CPC title

• B25J9/1664 Primary

  characterised by motion, path, trajectory planning · CPC title

• B25J13/003

  by means of an audio-responsive input (audible safety signals B25J19/061) · CPC title

• B25J13/089

  Determining the position of the robot with reference to its environment · CPC title