Interacting with machines using natural language input and a state graph

Patent metadata

Field	Value
Publication number	US-12320636-B2
Application number	US-202117518172-A
Country	US
Kind code	B2
Filing date	Nov 3, 2021
Priority date	Aug 10, 2021
Publication date	Jun 3, 2025
Grant date	Jun 3, 2025

Abstract

In one embodiment, a method is provided. The method includes obtaining sensor data indicative of a set of objects detected within an environment. The method also includes determining a set of positions of the set of objects and a set of properties of the set of objects based on the sensor data. The method further includes generating a state graph based on the sensor data. The state graph represents the set of objects and the set of positions of the set of objects. The state graph includes a set of object nodes to represent the set of objects and a set of property nodes to represent the set of properties of the set of objects. The state graph is provided to a graph enhancement module that updates the state graph with additional data to generate an enhanced state graph.

First claim

What is claimed is:

1. A method, comprising:
obtaining sensor data on a set of positions of a set of objects and a set of physical properties of the set of objects from a set of sensor devices, wherein the sensor data is indicative of the set of objects detected within an environment;
generating a state graph based on the sensor data, wherein:
the state graph represents the set of objects and the set of positions of the set of objects within the environment;
the state graph comprises a set of object nodes to represent the set of objects; and
the state graph comprises a set of property nodes to represent the set of physical properties of the set of objects, wherein the set of object nodes and the set of property nodes are interconnected with edges that respectively indicate transitions between different states;

receiving, via a user input module, user input that describes an objective to be achieved;
providing the state graph to a graph enhancement module to generate an enhanced state graph by:
adding a goal node to the state graph, wherein the goal node represents the objective that was described by the user input; and
inferring a new property of an identified object among the set of objects based on a correlation between the user input and an existing property among the set of physical properties, wherein the new property and the existing property are physical properties of the identified object;
adding an additional property node to the set of property nodes of the state graph, wherein the additional property node represents the new property of the identified object; and
adding or updating a relationship node to the state graph based on temporal relationships for the set of objects, wherein the temporal relationships indicate changes to distances between objects of the set of objects and changes of physical properties of the set of objects over a specified period of time;

generating instructions for a mechanical system based on the enhanced state graph; and
processing the instructions, via a processing device, to cause the mechanical system to interact with one or more of the set of objects within the environment to achieve the objective.

2. The method of claim 1 , further comprising:
determining a set of relationships between the set of objects based on one or more spatial calculi, wherein the set of relationships comprise spatial relationships between the set of objects.

3. The method of claim 2 , wherein determining the set of relationships between the set of objects comprises:
determining a relationship for each pair of objects in the set of objects.

4. The method of claim 2 , wherein the state graph further comprises a set of relationship nodes to represent the set of relationships between the set of objects.

5. The method of claim 2 , wherein determining the set of relationships between the set of objects comprises:
determining temporal information about the set of objects, wherein the temporal information is indicative of changes to one or more of the set of objects, the set of physical properties, and the set of positions, over a period of time.

6. The method of claim 5 , wherein the temporal information further represent the changes to one or more of the set of objects, the set of physical properties including temperature or color, and the set of positions in the environment, over the period of time.

7. The method of claim 1 , wherein each property node of the set of property nodes is connected to a respective object node of the set of object nodes in the state graph.

8. The method of claim 1 , wherein the set of positions of the set of objects and the set of properties of the set of objects are determined further based on one or more machine learning models.

9. An apparatus, comprising:
a memory to store a binary code of a controller; and
a processing device, operatively coupled to the memory, to:
obtain sensor data on a set of positions of a set of objects and a set of physical properties of the set of objects from a set of sensor devices, wherein the sensor data is indicative of the set of objects detected within an environment;
generate a state graph based on the sensor data, wherein:
the state graph represents the set of objects and the set of positions of the set of objects within the environment;
the state graph comprises a set of object nodes to represent the set of objects; and
the state graph comprises a set of property nodes to represent the set of physical properties of the set of objects, wherein the set of object nodes and the set of property nodes are interconnected with edges that respectively indicate transitions between different states;

receive, via a user input module, user input that describes an objective to be achieved;
provide the state graph to a graph enhancement module to generate an enhanced state graph, wherein the graph enhancement module is configured to:
add a goal node to the state graph, wherein the goal node represents the objective that was described by the user input;
infer a new property of an identified object among the set of objects based on a correlation between the user input and an existing property among the set of properties, wherein the new property and the existing property are physical properties of the identified object;
add an additional property node to the set of property nodes of the state graph, wherein the additional property node represents the new property of the identified object; and
add or update a relationship node to the state graph based on temporal relationships for the set of objects, wherein the temporal relationships indicate changes to distances between objects of the set of objects and changes of physical properties of the set of objects over a specified period of time;

generate instructions for a mechanical system based on the enhanced state graph; and
process the instructions, via a processing device, to cause the mechanical system to interact with one or more of the set of objects within the environment to achieve the objective.

10. The apparatus of claim 9 , wherein the processing device is further to:
determine a set of relationships between the set of objects based on one or more spatial calculi, wherein the set of relationships comprise spatial relationships between the set of objects.

11. The apparatus of claim 10 , wherein to determine the set of relationships between the set of objects the processing device is further to:
determine a relationship for each pair of objects in the set of objects.

12. The apparatus of claim 10 , wherein the state graph further comprises a set of relationship nodes to represent the set of relationships between the set of objects.

13. The apparatus of claim 10 , wherein to determine the set of relationships between the set of objects the processing device is further to:
determine temporal information about the set of objects, wherein the temporal information is indicative of changes to one or more of the set of objects, the set of physical properties, and the set of positions, over a period of time.

14. The apparatus of claim 13 , wherein the temporal information further represents the changes to one or more of the set of objects, the set of physical properties including temperature or color, and the set of positions in the environment, over the period of time.

15. The apparatus of claim 9 , wherein each property node of the set of property nodes is connected to a respective object node of the set of object nodes.

16. The apparatus of claim 9 , wherein the set of positions of the set of objects and the set of physical properties o

Assignees

• Palo Alto Res Ct Inc
• Xerox Corp

Inventors

• Mohan Shiwali
• Klenk Matthew
• Shreve Matthew
• Ang Aaron
• Maxwell Iii John Turner
• Evans Kent

Classifications

• B25J9/1664 Primary

  characterised by motion, path, trajectory planning · CPC title

• B25J9/163

  learning, adaptive, model based, rule based expert control · CPC title

• B25J13/089

  Determining the position of the robot with reference to its environment · CPC title

• B25J13/003

  by means of an audio-responsive input (audible safety signals B25J19/061) · CPC title

• G06F18/21375

  involving differential geometry, e.g. embedding of pattern manifold · CPC title