Object trajectory association and tracking

Patent metadata

Field	Value
Publication number	US-11577723-B2
Application number	US-202016938020-A
Country	US
Kind code	B2
Filing date	Jul 24, 2020
Priority date	Jun 29, 2020
Publication date	Feb 14, 2023
Grant date	Feb 14, 2023

Abstract

Systems, device, and methods for trajectory association and tracking are provided. A method can include obtaining input data indicative of a respective trajectory for each of one or more first objects for a first time step and input data indicative of a respective trajectory for each of one or more second objects for a second time step subsequent to the first time step. The method can include generating, using a machine-learned model, a temporally-consistent trajectory for at least one of the one or more first objects or the one or more second objects based at least in part on the input data and determining a third predicted trajectory for the at least one of the one or more first objects or the one or more second objects for at least the second time step based at least in part on the temporally-consistent trajectory.

First claim

What is claimed is:

1. A computer-implemented method for generating a trajectory for an object, comprising:
obtaining, by a computing system comprising one or more computing devices, input data indicative of a respective trajectory for each of one or more first objects at a first time step;
obtaining, by the computing system, input data indicative of a respective trajectory for each of one or more second objects at a second time step, the second time step subsequent to the first time step and the one or more second objects comprising at least one of the one or more first objects;
generating, by the computing system using a machine-learned model, a temporally-consistent trajectory for at least one of the one or more first objects or the one or more second objects based at least in part on the input data indicative of the respective trajectory for the one or more first objects and the input data indicative of the respective trajectory for the one or more second objects; and
determining, by the computing system, a third predicted trajectory for the at least one of the one or more first objects or the one or more second objects for at least the second time step based at least in part on the temporally-consistent trajectory for the at least one of the one or more first objects or the one or more second objects;
wherein generating, by the computing system using the machine-learned model, the temporally-consistent trajectory for the at least one of the one or more first objects or the one or more second objects comprises:
generating, by the machine-learned model, association data comprising data descriptive of an association of each respective trajectory of a first object of the one or more first objects to each respective trajectory of a second object of the one or more second objects; and
generating, by the machine-learned model, the temporally-consistent trajectory for at least one of the one or more second objects based at least in part on the association data.

2. The computer-implemented method of claim 1 , wherein the at least one of the one or more first objects or the one or more second objects comprises an occluded object; and
wherein generating, by the computing system using the machine-learned model, the temporally-consistent trajectory for the at least one of the one or more first objects or the one or more second objects comprises:
propagating a previous trajectory associated with the occluded object forward in time.

3. The computer-implemented method of claim 1 , wherein generating, by the computing system using the machine-learned model, the temporally-consistent trajectory for the at least one of the one or more first objects or the one or more second objects comprises:
generating, by the machine-learned model, association data by the machine-learned model, the association data comprising data descriptive of an association of each respective trajectory for the one or more first objects to each respective trajectory for the one or more second objects; and
generating, by the machine-learned model, the temporally-consistent trajectory for the at least one of the one or more second objects based at least in part on the association data.

4. The computer-implemented method of claim 1 , wherein generating, by the computing system using the machine-learned model, the temporally-consistent trajectory for the at least one of the one or more first objects or the one or more second objects comprises refining, by the machine-learned model, the respective trajectory for the at least one of the one or more first objects or the one or more second objects.

5. The computer-implemented method of claim 1 , wherein the at least one object comprises an object of the one or more second objects that is not highly associated with an object in the one or more first objects; and
wherein generating, by the computing system using the machine-learned model, the temporally-consistent trajectory for the at least one of the one or more first objects or the one or more second objects comprises:
determining, by the computing system, to birth the temporally-consistent trajectory for the at least one of the one or more second objects.

6. The computer-implemented method of claim 1 , wherein the at least one object comprises an object of the one or more first objects that is not highly associated with an object in the one or more second objects; and
wherein generating, by the computing system using the machine-learned model, the temporally-consistent trajectory for the at least one of the one or more first objects or the one or more second objects comprises:
determining, by the computing system, to reap the temporally-consistent trajectory for the at least one of the one or more second objects.

7. The computer-implemented method of claim 1 , wherein the input data indicative of the first predicted trajectory or the input data indicative of the second predicted trajectory comprises data output from a first stage model configured to perceive the one or more first objects or the one or more second objects and predict a trajectory for the one or more first objects or the one or more second objects based at least in part on sensor data obtained by one or more sensors of an autonomous vehicle.

8. The computer-implemented method of claim 1 , wherein the input data indicative of the first predicted trajectory or the input data indicative of the second predicted trajectory comprises one or more trajectories, bounding boxes, probabilities, feature maps, identifiers, and/or tracked or untracked higher order states for the one or more first objects or one or more second objects.

9. The computer-implemented method of claim 1 , wherein the first time step comprises a past time step and wherein the second time step comprises a current time step.

10. The computer-implemented method of claim 1 , wherein the machine-learned model has been trained using a loss function comprising an association loss parameter and one or more trajectory loss parameters.

11. The computer-implemented method of claim 1 , wherein the machine-learned model has been trained for one or more training steps using one or more ground-truth temporally-consistent trajectories and trained for one or more training steps using one or more outputs from a first stage model configured to determine a predicted trajectory for one or more objects using sensor data.

12. The computer-implemented method of claim 1 , wherein the machine-learned model comprises a LSTM, GRU or RNN model.

13. An autonomous vehicle, comprising:
one or more processors;
a memory comprising one or more tangible non-transitory computer-readable media, the media storing computer-readable instructions that when executed by the one or more processors cause the one or more processors to perform operations comprising:
obtaining input data indicative of a respective trajectory for each of one or more first objects for a first time step from a first stage model configured to generate predicted trajectories for detected objects using sensor data from sensors of the autonomous vehicle obtained at the first time step;
obtaining input data indicative of a respective trajectory for each of one or more second objects for a second time step from the first stage model using sensor data from the sensors of the autonomous vehicle obtained at the second time step, the second time step subsequent to the first time step;
generating a temporally-consistent trajectory for at least one of the one or more second objects based at least in part on the input data indicative of the respective trajectory for the one or more first objects and the input data indicative of the respective traj

Assignees

• Uatc Llc

Inventors

• Gautam Shivam
• Zhang sida
• Meyer Gregory P
• Vallespi-Gonzalez Carlos
• Becker Brian C

Classifications

• B60W40/04

  Traffic conditions · CPC title

• G06N3/045

  Combinations of networks · CPC title

• B60W2554/4042

  Longitudinal speed · CPC title

• G06N3/084 Primary

  Backpropagation, e.g. using gradient descent · CPC title

• B60W2554/4044

  Direction of movement, e.g. backwards · CPC title