End-to-end tracking of objects

Patent metadata

Field	Value
Publication number	US-11017550-B2
Application number	US-201816122203-A
Country	US
Kind code	B2
Filing date	Sep 5, 2018
Priority date	Nov 15, 2017
Publication date	May 25, 2021
Grant date	May 25, 2021

Abstract

Systems and methods for detecting and tracking objects are provided. In one example, a computer-implemented method includes receiving sensor data from one or more sensors. The method includes inputting the sensor data to one or more machine-learned models including one or more first neural networks configured to detect one or more objects based at least in part on the sensor data and one or more second neural networks configured to track the one or more objects over a sequence of sensor data. The method includes generating, as an output of the one or more first neural networks, a 3D bounding box and detection score for a plurality of object detections. The method includes generating, as an output of the one or more second neural networks, a matching score associated with pairs of object detections. The method includes determining a trajectory for each object detection.

First claim

What is claimed is:

1. A computer-implemented method of detecting objects of interest, comprising:
receiving, by a computing system comprising one or more computing devices, sensor data from one or more sensors configured to generate sensor data associated with an environment;
inputting, by the computing system, the sensor data to one or more machine-learned models including one or more first neural networks configured to detect one or more objects in the environment based at least in part on the sensor data and one or more second neural networks configured to track the one or more objects over a sequence of sensor data inputs;
generating, by the computing system as an output of the one or more first neural networks, a three-dimensional (3D) bounding box and detection score for each of a plurality of object detections;
generating, by the computing system as an output of the one or more second neural networks, a matching score associated with object detections over the sequence of sensor data inputs;
generating, by the computing system using a flow network, a flow graph based at least in part on the matching scores; and
determining, by the computing system using a linear program, a trajectory for each object detection based at least in part on the matching scores associated with the object detections over the sequence of sensor data inputs, wherein determining the trajectory for each object detection is based at least in part on the flow graph.

2. The computer-implemented method of claim 1 , wherein:
determining the trajectory for each object detection comprises determining the trajectory for each object detection based on one or more linear constraints.

3. The computer-implemented method of claim 1 , wherein:
generating a 3D bounding box comprises generating the 3D bounding box for each object detection based on sensor data including one or more LIDAR pointclouds from one or more first sensors; and
generating a detection score for each object detection comprises generating the detection score based on red, green, blue (RGB) data from one or more second sensors.

4. The computer-implemented method of claim 1 , wherein:
generating, as an output of the one or more first neural networks, the 3D bounding box and detection score for each object detection is based on a predetermined optimization of the one or more second neural networks.

5. The computer-implemented method of claim 1 , wherein:
generating, as an output of the one or more first neural networks, the detection score for each object detection comprises generating the detection score based on optimization of the one or more second neural networks for generating matching scores.

6. The computer-implemented method of claim 1 , wherein:
the one or more machine-learned models are trained end-to-end by training the one or more first neural networks and the one or more second neural networks based at least in part on detected errors of trajectories generated using the linear program during training.

7. The computer-implemented method of claim 1 , wherein the one or more sensors are configured to generate sensor data relative to an autonomous vehicle:
the method further comprises generating, by the computing system, one or more vehicle control signals for the autonomous vehicle based at least in part on trajectories for the object detections.

8. The computer-implemented method of claim 1 , wherein:
the one or more first neural networks are one or more first convolutional neural networks; and
the one or more second neural networks are one or more second convolutional neural networks.

9. The computer-implemented method of claim 1 , wherein:
the sensor data input to the one or more machine-learned models includes first image data and second image data, the first image data representing an earlier point in time than the second image data.

Assignees

• Uatc Llc

Inventors

• Frossard Davi Eugenio Nascimento
• Urtasun Raquel

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/08

  Learning methods · CPC title

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06T7/20 Primary

  Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51) · CPC title