Geo-motion and appearance aware data association

Patent metadata

Field	Value
Publication number	US-11830203-B2
Application number	US-202217728682-A
Country	US
Kind code	B2
Filing date	Apr 25, 2022
Priority date	Dec 23, 2019
Publication date	Nov 28, 2023
Grant date	Nov 28, 2023

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for associating a new measurement of an object surrounding a vehicle with a maintained track. One of the methods includes receiving an object track for a particular object, receiving a new measurement characterizing a new object at a new time step, and determining whether the new object is the same as the particular object, comprising: generating a representation of the new object at the new and preceding time steps; generating a representation of the particular object at the new and preceding time steps; processing a first network input comprising the representations using a first neural network to generate an embedding of the first network input; and processing the embedding of the first network input using a second neural network to generate a predicted likelihood that the new object and the particular object are the same.

First claim

What is claimed is:

1. A method comprising:
receiving an object track for a particular object that identifies a location in an environment of the particular object at a preceding time step;
receiving a new measurement characterizing a location in the environment of a new object at a new time step; and
determining whether the new object is the same as the particular object, comprising:
generating, from the new measurement, a first representation that identifies a predicted location in the environment of the new object at the preceding time step, wherein the generating is based on the location of the new object at the new time step;
generating, from the object track, a second representation that identifies the location of the particular object at the preceding time step; and
processing a network input generated from at least the first representation and the second representation using a neural network to generate a predicted likelihood that the new object and the particular object are the same.

2. The method of claim 1 , wherein the network input has further been generated from one or more of:
a third representation generated from the new measurement that identifies the location of the new object at the new time step; or
a fourth representation generated from the object track that identifies a predicted location in the environment of the particular object at the new time step.

3. The method of claim 1 , wherein one or more of the first representation or second representation comprises a top-down rendered image.

4. The method of claim 3 , wherein the top-down rendered image for at least one of the representations comprises a plurality of pixels depicting a bounding box of the respective object at the respective time step.

5. The method of claim 4 , wherein a plurality of pixel values of the respective pixels depicting the bounding box characterize one or more of (i) a direction of motion of the respective object at the respective time step, or (ii) a magnitude of a velocity of the respective object at the respective time step.

6. The method of claim 3 , wherein a center position of the top-down rendered image for at least one of the representations characterizes the location of the new object at the new time step.

7. The method of claim 1 , wherein:
one or more of the first representation or second representation comprises a numerical feature representation of the respective object at the respective time step, the numerical feature representation comprising one or more location values and one or more motion values of the respective object at the respective time step.

8. The method of claim 1 , wherein:
the method further comprises identifying one or more of: a first environment measurement of the environment at the new time step or a second environment measurement of the environment at the preceding time step; and
the network input has further been generated from one or more of:
a fifth representation generated from the new measurement that represents the environment at the new time step; and
a sixth representation generated from the object track that represents the environment at the preceding time step.

9. The method of claim 1 , wherein:
the method further comprises identifying one or more of: a first perspective image characterizing the new object at the new time step, or a second perspective image characterizing the particular object at the preceding time step; and
the network input has further been generated from one or more of the first perspective image or the second perspective image.

10. The method of claim 1 , wherein determining whether the new object is the same as the particular object further comprises:
receiving one or more other object tracks for respective other objects;
for each other object, generating a respective predicted likelihood that the new object and the other object are the same; and
determining, using the predicted likelihoods, whether the particular object and the new object are the same.

11. A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
receiving an object track for a particular object that identifies a location in an environment of the particular object at a preceding time step;
receiving a new measurement characterizing a location in the environment of a new object at a new time step; and
determining whether the new object is the same as the particular object, comprising:
generating, from the new measurement, a first representation that identifies a predicted location in the environment of the new object at the preceding time step, wherein the generating is based on the location of the new object at the new time step;
generating, from the object track, a second representation that identifies the location of the particular object at the preceding time step; and
processing a network input generated from at least the first representation and the second representation using a neural network to generate a predicted likelihood that the new object and the particular object are the same.

12. The system of claim 11 , wherein the network input has further been generated from one or more of:
a third representation generated from the new measurement that identifies the location of the new object at the new time step; or
a fourth representation generated from the object track that identifies a predicted location in the environment of the particular object at the new time step.

13. The system of claim 11 , wherein one or more of the first representation or second representation comprises a top-down rendered image.

14. The system of claim 11 , wherein:
one or more of the first representation or second representation comprises a numerical feature representation of the respective object at the respective time step, the numerical feature representation comprising one or more location values and one or more motion values of the respective object at the respective time step.

15. The system of claim 11 , wherein determining whether the new object is the same as the particular object further comprises:
receiving one or more other object tracks for respective other objects;
for each other object, generating a respective predicted likelihood that the new object and the other object are the same; and
determining, using the predicted likelihoods, whether the particular object and the new object are the same.

16. One or more non-transitory computer storage media encoded with computer program instructions that when executed by a plurality of computers cause the plurality of computers to perform operations comprising:
receiving an object track for a particular object that identifies a location in an environment of the particular object at a preceding time step;
receiving a new measurement characterizing a location in the environment of a new object at a new time step; and
determining whether the new object is the same as the particular object, comprising:
generating, from the new measurement, a first representation that identifies a predicted location in the environment of the new object at the preceding time step, wherein the generating is based on the location of the new object at the new time step;
generating, from the object track, a second representation that identifies the location of the particular object at the preceding time step; and
processing a network input generated from at least the first representation and the second representation using a neural net

Assignees

• Waymo Llc

Inventors

• Yu ruichi
• Hemachandra Sachithra Madhawa
• Mahon Ian James
• Li Congcong

Classifications

• G05D1/0088

  characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours (using knowledge based models G06N5/00) · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06T7/248 Primary

  involving reference images or patches · CPC title

• G06N3/045

  Combinations of networks · CPC title