Object association using machine learning models

Patent metadata

Field	Value
Publication number	US-11766783-B2
Application number	US-202217817076-A
Country	US
Kind code	B2
Filing date	Aug 3, 2022
Priority date	Dec 17, 2019
Publication date	Sep 26, 2023
Grant date	Sep 26, 2023

Abstract

A method includes receiving sensor data representing a first object in an environment and generating, based on the sensor data, a first state vector that represents physical properties of the first object. The method also includes generating, by a first machine learning model and based on the first state vector and a second state vector that represents physical properties of a second object previously observed in the environment, a metric indicating a likelihood that the first object is the same as the second object. The method further includes determining, based on the metric, to update the second state vector and updating, by a second machine learning model configured to maintain the second state vector over time and based on the first state vector, the second state vector to incorporate into the second state vector information concerning physical properties of the second object as represented in the first state vector.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving sensor data representing a first object in an environment;
generating, based on the sensor data, a first state vector that represents physical properties of the first object;
generating, for each respective object of a plurality of objects previously observed in the environment, a corresponding metric indicating a likelihood that the first object is the same as the respective object, wherein the corresponding metric is generated by a first machine learning (ML) model and based on (i) the first state vector and (ii) a respective state vector that represents physical properties of the respective object;
determining, based on the corresponding metric of each respective object of the plurality of objects, that the first object is different from each respective object of the plurality of objects; and
based on determining that the first object is different from each respective object of the plurality of objects, initializing a second ML model configured to represent the physical properties of the first object over time, wherein the second ML model is initialized based on the first state vector.

2. The computer-implemented method of claim 1 , wherein the sensor data is received from a sensor on a robotic device.

3. The computer-implemented method of claim 1 , wherein determining that the first object is different from each respective object of the plurality of objects comprises:
determining that the corresponding metric of each respective object of the plurality of objects does not exceed a threshold value; and
based on determining that the corresponding metric of each respective object of the plurality of objects does not exceed a threshold value, determining that the first object is different from each respective object of the plurality of objects.

4. The computer-implemented method of claim 1 , wherein the plurality of objects previously observed in the environment comprises every object for which the robotic device stores a corresponding ML model that maintains a corresponding state vector over time.

5. The computer-implemented method of claim 1 , wherein the plurality of objects previously observed in the environment comprises n objects associated with corresponding n state vectors that are nearest to the first state vector.

6. The computer-implemented method of claim 1 , further comprising:
receiving additional sensor data representing a second object in the environment;
generating, based on the additional sensor data, a second state vector that represents physical properties of the second object;
generating, by the first ML model and based on (i) the first state vector and (ii) the second state vector, an additional metric indicating a likelihood that the first object is the same as the second object;
determining, based on the additional metric, to update the first state vector; and
updating the first state vector by the second ML model based on the second state vector.

7. The computer-implemented method of claim 6 , wherein determining to update the first state vector comprises:
determining, based on the additional metric, that the second object as represented by the second state vector is the same as the first object represented by the first state vector; and
based on determining that the second object as represented by the second state vector is the same as the first object represented by the first state vector, determining to update the second state vector.

8. The computer-implemented method of claim 7 , wherein determining that the second object as represented by the second state vector is the same as the first object represented by the first state vector comprises one or more of:
determining that the additional metric is a highest metric of a plurality of metrics generated by the first ML model based on (i) the second state vector and (ii) each of a plurality of other state vectors; or
determining that the additional metric exceeds a threshold value.

9. The computer-implemented method of claim 1 , wherein the second ML model comprises an instance of a class-specific type of ML model that corresponds to a class of the first object.

10. The computer-implemented method of claim 1 , wherein the second ML model comprises a long short-term memory neural network.

11. The computer-implemented method of claim 1 , wherein the first state vector comprises a plurality of values indicating one or more of: (i) a position of the first object within the environment, (ii) a size of the first object, (iii) a classification of the first object, (iv) an embedding representing the first object, (v) a time at which the first state vector was last updated, (vi) a distance between the sensor and the first object, (vii) a confidence with which the first state vector represents the physical properties of the first object, (viii) an indication of whether the first object is within a current field of view of the sensor, (ix) a weight of the first object, or (x) a time at which the first object was last observed by the sensor.

12. The computer-implemented method of claim 1 , wherein the first ML model and the second ML model are trained using a loss function that interrelates outputs of the first ML model and the second ML model and is configured to maximize a confidence of the first ML model when the first ML model correctly determines, based on a respective state vector maintained by the second ML model and a training state vector, whether the respective state vector and the training state vector represent two different objects or the same object.

13. The computer-implemented method of claim 1 , wherein at least one of (i) the corresponding metric or (ii) the first state vector is used to select parameters of a Kalman filter used by an object tracker of the robotic device.

14. The method of claim 1 , further comprising:
determining, based on the first state vector, one or more operations for the robotic device to perform to interact with the first object.

15. A system comprising:
a processor; and
a non-transitory computer-readable storage medium having stored thereon instructions that, when executed by the processor, cause the processor to perform operations comprising:
receiving sensor data representing a first object in an environment;
generating, based on the sensor data, a first state vector that represents physical properties of the first object;
generating, for each respective object of a plurality of objects previously observed in the environment, a corresponding metric indicating a likelihood that the first object is the same as the respective object, wherein the corresponding metric is generated by a first machine learning (ML) model and based on (i) the first state vector and (ii) a respective state vector that represents physical properties of the respective object;
determining, based on the corresponding metric of each respective object of the plurality of objects, that the first object is different from each respective object of the plurality of objects; and
based on determining that the first object is different from each respective object of the plurality of objects, initializing a second ML model configured to represent the physical properties of the first object over time, wherein the second ML model is initialized based on the first state vector.

16. The system of claim 15 , wherein determining that the first object is different from each respective object of the plurality of objects comprises:
determining that the corresponding metric of each respective object of the plurality of objects does not ex

Assignees

• Google Llc

Inventors

• Kirmani Sean
• Satat Guy
• Quinlan Michael

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• B25J9/1697 Primary

  Vision controlled systems · CPC title

• B25J9/161

  Hardware, e.g. neural networks, fuzzy logic, interfaces, processor · CPC title

• G06N3/04

  Architecture, e.g. interconnection topology · CPC title