Perception and motion prediction for autonomous devices

Patent metadata

Field	Value
Publication number	US-11548533-B2
Application number	US-202016826895-A
Country	US
Kind code	B2
Filing date	Mar 23, 2020
Priority date	Mar 23, 2019
Publication date	Jan 10, 2023
Grant date	Jan 10, 2023

Abstract

Systems, methods, tangible non-transitory computer-readable media, and devices associated with object perception and prediction of object motion are provided. For example, a plurality of temporal instance representations can be generated. Each temporal instance representation can be associated with differences in the appearance and motion of objects over past time intervals. Past paths and candidate paths of a set of objects can be determined based on the temporal instance representations and current detections of objects. Predicted paths of the set of objects using a machine-learned model trained that uses the past paths and candidate paths to determine the predicted paths. Past path data that includes information associated with the predicted paths can be generated for each object of the set of objects respectively.

First claim

What is claimed is:

1. A computer-implemented method of perception and motion forecasting, the computer-implemented method comprising:
generating a plurality of temporal instance representations respectively associated with differences in an appearance and a motion of one or more objects over past time intervals, wherein the plurality of temporal instance representations respectively comprise one or more appearance features of the one or more objects at the past time intervals and one or more motion features of one or more objects at the past time intervals;
determining, based at least in part on the plurality of temporal instance representations and current detections of a plurality of objects comprising the one or more objects, one or more past paths of the one or more objects over the past time intervals and one or more candidate paths of the plurality of objects over a plurality of time intervals comprising a current time interval and at least one of the past time intervals;
determining one or more predicted paths of the plurality of objects based at least in part on one or more machine-learned models, the one or more machine-learned models utilizing the one or more past paths and the one or more candidate paths to infer the one or more predicted paths; and
generating path data comprising information associated with the one or more predicted paths for the plurality of objects respectively.

2. The computer-implemented method of claim 1 , wherein the one or more past paths comprise at least one null path, and wherein the determining based at least in part on the plurality of temporal instance representations and the current detections of the plurality of objects comprising the one or more objects comprises:
determining, based at least in part on one or more comparisons of the plurality of objects to the one or more objects, whether the plurality of objects includes one or more newly detected objects not included in the one or more objects from the past time intervals; and
associating the one or more newly detected objects with the at least one null path.

3. The computer-implemented method of claim 1 , wherein the generating the plurality of temporal instance representations comprises:
obtaining data associated with the motion of the one or more objects over the past time intervals from an object path memory; and
obtaining data associated with the appearance of the one or more objects over the past time intervals from an appearance memory that is different from the object path memory.

4. The computer-implemented method of claim 1 , wherein the generating the plurality of temporal instance representations comprises:
generating based at least in part on a plurality of machine-learned feature extraction models and multi-sensor data, a plurality of feature maps associated with the appearance and the motion of the one or more objects over the one or more past time intervals, wherein the multi-sensor data is based at least in part on sensor outputs from a plurality of different types of sensors; and
generating the plurality of temporal instance representations based at least in part on the plurality of feature maps.

5. The computer-implemented method of claim 4 , wherein the multi-sensor data comprises one or more light detection and ranging (LiDAR) sweeps, map data comprising information associated with one or more locations in an environment comprising the one or more objects, or one or more images comprising the one or more objects.

6. The computer-implemented method of claim 1 , wherein the plurality of temporal instance representations respectively comprise a concatenation of one or more appearance features and one or more motion features respectively associated with the appearance and the motion of the one or more objects over the past time intervals.

7. The computer-implemented method of claim 1 , wherein a number of the one or more candidate paths is at least as great as a combination of a number of the one or more past paths and a number of the current detections of the plurality of objects.

8. The computer-implemented method of claim 1 , wherein the determining the one or more predicted paths of the plurality of objects based at least in part on one or more machine-learned models comprises:
determining a plurality of matching scores corresponding to the plurality of temporal instance representations, wherein the plurality of matching scores are respectively based at least in part on differences between the appearance and the motion of the plurality of objects over the one or more past paths and the appearance and the motion of the plurality of objects over the one or more candidate paths; and
determining the one or more predicted paths based at least in part on the plurality of matching scores associated with a least amount of difference in the appearance and the motion of the plurality of objects.

9. The computer-implemented method of claim 1 , wherein the one or more machine-learned models are configured to respectively compare the appearance and the motion of the plurality of objects along the one or more past paths at the past time intervals to the appearance and the motion of the plurality of objects along the one or more candidate paths at the past time intervals.

10. The computer-implemented method of claim 1 , wherein the one or more machine-learned models are trained based at least in part on minimization of a loss associated with one or more differences between one or more predicted training paths and one or more ground-truth paths, wherein the one or more predicted training paths are generated using training data and the one or more machine-learned models, and wherein the training data comprises a plurality of training temporal instance representations and a plurality of training object detections.

11. The computer-implemented method of claim 10 , wherein the loss is based at least in part on a loss function associated with a detection loss, a matching loss, a confidence score loss, a refinement loss, or a prediction loss.

12. The computer-implemented method of claim 10 , wherein the loss is inversely correlated with similarity of the one or more predicted training paths relative to the one or more ground-truth paths.

13. The computer-implemented method of claim 1 , wherein the determining the one or more predicted paths of the plurality of objects based at least in part on one or more machine-learned models comprises:
determining for the one or more candidate paths, and based at least in part on the plurality of temporal instance representations and the one or more machine-learned models comprising a machine-learned refinement model, one or more confidence scores, one or more path refinements, and one or more candidate predicted paths;
generating one or more refined candidate paths based at least in part on the one or more candidate predicted paths and the one or more path refinements;
ranking the one or more refined candidate paths based at least in part on the one or more confidence scores; and
determining the one or more predicted paths based at least in part on the ranking of the one or more refined candidate paths.

14. The computer-implemented method of claim 13 , wherein the one or more confidence scores are associated with a respective estimated accuracy of the one or more candidate predicted paths, and wherein the one or more path refinements comprise adjustments of bounding boxes associated with the appearance of the plurality of objects along the one or more candidate paths.

15. A computing system comprising:
one or more processors;
a memory comprising one or more tangible non-tra

Assignees

• Uatc Llc

Inventors

• Liang Ming
• Yang Bin
• Chen Yun
• Urtasun Raquel

Classifications

• B60W60/00272 Primary

  relying on extrapolation of current movement · CPC title

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N20/00

  Machine learning · CPC title

• G01S17/89

  for mapping or imaging · CPC title

• G06N5/04

  Inference or reasoning models · CPC title