System for interacting with machines using natural language input

Patent metadata

Field	Value
Publication number	US-12031814-B2
Application number	US-202117518429-A
Country	US
Kind code	B2
Filing date	Nov 3, 2021
Priority date	Aug 10, 2021
Publication date	Jul 9, 2024
Grant date	Jul 9, 2024

Abstract

A method is provided. The method includes obtaining sensor data indicative of a set of objects detected within an environment. The method also includes generating a state graph based on the sensor data. The state graph includes a set of object nodes and a set of property nodes. The method further includes obtaining user input data generated based on a natural language input. The method further includes updating the state graph based on the user input data to generate an enhanced state graph. The enhanced state graph includes additional nodes generated based on the user input data. The method further includes generating a set of instructions for a set of mechanical systems based on the enhanced state graph. The method further includes operating the set of mechanical systems to achieve a set of objectives based on the set of instructions.

First claim

What is claimed is:

1. A method, comprising:
obtaining sensor data from a set of imaging sensor devices, wherein the sensor data comprises image data indicative of a set of objects detected within an environment in proximity to a set of mechanical systems configured to interact with the set of objects;
generating a state graph based on the sensor data, wherein:
the state graph represents the set of objects and a set of positions of the set of objects within the environment;
the state graph comprises a set of object nodes to represent the set of objects; and
the state graph comprises a set of property nodes to represent a set of properties of the set of objects;

obtaining user input data, wherein the user input data is generated based on a natural language input; and
updating the state graph based on the user input data to generate an enhanced state graph, wherein the enhanced state graph comprises the set of object nodes, the set of property nodes, and a goal node generated based on the user input data, wherein the goal node indicates an objective;
generating a set of instructions for the set of mechanical systems based on the enhanced state graph; and
operating the set of mechanical systems to interact with one or more of the set of objects within the environment based on the set of instructions;
after operating the set of mechanical systems, obtaining additional sensor data from the set of imaging sensor devices and updating the set of property nodes of the enhanced state graph to generate an updated enhanced state graph based on the additional sensor data to indicate changes to the set of properties of the set of objects caused by the set of mechanical systems;
determining whether the objective has been met based on the updated enhanced state graph; and
responsive to determining that the objective has not been met, generating a new set of instructions for the set of mechanical systems based on the updated enhanced state graph.

2. The method of claim 1 , further comprising:
determining a set of relationships between the set of objects based on one or more spatial calculi, wherein the set of relationships comprise spatial relationships between the set of objects.

3. The method of claim 2 , wherein the state graph further comprises a set of relationship nodes to represent the set of relationships between the set of objects.

4. The method of claim 1 , wherein the set of positions of the set of objects and the set of properties of the set of objects are determined further based on one or more machine learning models.

5. The method of claim 1 , wherein:
the user input data indicates the set of objectives associated with one or more objects of the set of objects.

6. The method of claim 5 wherein updating the state graph comprises:
generating a set of goal nodes, wherein the set of goal nodes represent the set of objectives associated with the one or more objects of the set of objects; and
the additional nodes comprise the set of goal nodes.

7. The method of claim 1 , wherein updating the state graph comprises:
determining a set of additional properties of the set of objects based on the user input data; and
generating a set of additional property nodes to represent the set of additional properties.

8. The method of claim 1 , further comprising:
identifying a set of locations for the subset of the set of objects; and
identifying a subset of the set of mechanical systems based on the set of locations.

9. The method of claim 8 , wherein generating the set of instructions comprises:
generating the set of instructions for the subset of the set of mechanical systems.

10. The method of claim 6 , wherein generating the set of instructions is further based on the set of goal nodes of the enhanced state graph.

11. The method of claim 6 , further comprising:
determining a portion of the environment based on the set of goal nodes, wherein the portion of the environment is associated with the set of objectives; and
determining whether the set of instructions cause the set of mechanical systems to operate within the portion of the environment.

12. The method of claim 11 , further comprising:
in response to determining that the set of instructions do not cause the set of mechanical systems to operate within the portion of the environment, generating a second set of instructions for the set of mechanical systems based on the enhanced state graph.

13. An apparatus, comprising:
a memory to store a binary code of a controller; and
a processing device, operatively coupled to the memory, to:
obtain sensor data from a set of imaging sensor devices, wherein the sensor data comprises image data indicative of a set of objects detected within an environment in proximity to a set of mechanical systems configured to interact with the set of objects;
generate a state graph based on the sensor data, wherein:
the state graph represents the set of objects and a set of positions of the set of objects within the environment;
the state graph comprises a set of object nodes to represent the set of objects; and
the state graph comprises a set of property nodes to represent a set of properties of the set of objects;

obtain user input data, wherein the user input data is generated based on a natural language input; and
update the state graph based on the user input data to generate an enhanced state graph, wherein the enhanced state graph comprises the set of object nodes, the set of property nodes, and a goal node generated based on the user input data, wherein the goal node indicates an objective;
generate a set of instructions for the set of mechanical systems based on the enhanced state graph; and

operate the set of mechanical systems to interact with one or more of the set of objects within the environment based on the set of instructions;
after operating the set of mechanical systems, obtain additional sensor data from the set of imaging sensor devices and update the set of property nodes of the enhanced state graph to generate an updated enhanced state graph based on the additional sensor data to indicate changes to the set of properties of the set of objects caused by the set of mechanical systems;
determine whether the objective has been met based on the updated enhanced state graph; and
responsive to determining that the objective has not been met, generate a new set of instructions for the set of mechanical systems based on the updated enhanced state graph.

14. The apparatus of claim 13 , wherein the processing device is further to:
determine a set of relationships between the set of objects based on one or more spatial calculi, wherein the set of relationships comprise spatial relationships between the set of objects.

15. The apparatus of claim 14 , wherein the state graph further comprises a set of relationship nodes to represent the set of relationships between the set of objects.

16. The method of claim 13 , wherein:
the user input data indicates the set of objectives associated with one or more objects of the set of objects.

17. The apparatus of claim 16 wherein to update the state graph the processing device is further to:
generates a set of goal nodes, wherein the set of goal nodes represent the set of objectives associated with the one or more objects of the set of objects; and
the additional nodes comprise the set of goal nodes.

18. The method of claim 1 , wherein to update the state graph the processing device is further to:
determining a set of additional properties of the set of objects based on the user

Assignees

• Palo Alto Res Ct Inc
• Xerox Corp

Inventors

• Mohan Shiwali
• Klenk Matthew
• Shreve Matthew
• Ang Aaron
• Maxwell Iii John Turner
• Evans Kent

Classifications

• B25J9/1664 Primary

  characterised by motion, path, trajectory planning · CPC title

• B25J9/163

  learning, adaptive, model based, rule based expert control · CPC title

• B25J13/089

  Determining the position of the robot with reference to its environment · CPC title

• B25J13/003

  by means of an audio-responsive input (audible safety signals B25J19/061) · CPC title

• G06F18/21375

  involving differential geometry, e.g. embedding of pattern manifold · CPC title