Systems and methods for predicting the trajectory of an object with the aid of a location-specific latent map

Patent metadata

Field	Value
Publication number	US-11427210-B2
Application number	US-202016835408-A
Country	US
Kind code	B2
Filing date	Mar 31, 2020
Priority date	Sep 13, 2019
Publication date	Aug 30, 2022
Grant date	Aug 30, 2022

Abstract

Systems and methods for predicting the trajectory of an object are disclosed herein. One embodiment receives sensor data that includes a location of the object in an environment of the object; accesses a location-specific latent map, the location-specific latent map having been learned together with a neural-network-based trajectory predictor during a training phase, wherein the neural-network-based trajectory predictor is deployed in a robot; inputs, to the neural-network-based trajectory predictor, the location of the object and the location-specific latent map, the location-specific latent map providing, to the neural-network-based trajectory predictor, a set of location-specific biases regarding the environment of the object; and outputs, from the neural-network-based trajectory predictor, a predicted trajectory of the object.

First claim

What is claimed is:

1. A system for predicting a trajectory of an object, the system comprising:
one or more sensors;
one or more processors; and
a memory communicably coupled to the one or more processors and storing:
a prediction module including instructions that when executed by the one or more processors cause the one or more processors to:
receive sensor data from the one or more sensors that includes a location of the object in an environment of the object;
access a location-specific latent map, the location-specific latent map having been learned together with a neural-network-based trajectory predictor during a training phase, wherein the neural-network-based trajectory predictor is deployed in a robot; and
input, to the neural-network-based trajectory predictor, the location of the object and the location-specific latent map, the location-specific latent map providing, to the neural-network-based trajectory predictor, a set of location-specific biases regarding the environment of the object; and

an output module including instructions that when executed by the one or more processors cause the one or more processors to output, from the neural-network-based trajectory predictor, a predicted trajectory of the object.

2. The system of claim 1 , wherein the robot is an ego vehicle and the object is a road agent external to the ego vehicle.

3. The system of claim 2 , wherein the ego vehicle is an autonomous vehicle.

4. The system of claim 2 , wherein the predicted trajectory of the object includes information regarding roadway geometries inferred by the neural-network-based trajectory predictor.

5. The system of claim 1 , wherein the robot and the object are one and the same thing and the robot is an ego vehicle that is at least partially controlled by a human driver.

6. The system of claim 5 , wherein the predicted trajectory of the object includes information regarding roadway geometries inferred by the neural-network-based trajectory predictor.

7. The system of claim 1 , wherein the set of location-specific biases regarding the environment of the object pertains to both visual and non-visual features of the environment of the object.

8. The system of claim 1 , further comprising an encoding module including instructions that when executed by the one or more processors cause the one or more processors to encode the location-specific latent map using a convolutional neural network (CNN).

9. The system of claim 1 , wherein the location-specific latent map corresponds to at least a portion of the environment of the object.

10. The system of claim 1 , wherein the neural-network-based trajectory predictor includes one of a social generative adversarial network (S-GAN) and a social long short-term memory (Social-LSTM) network.

11. A non-transitory computer-readable medium for predicting a trajectory of an object and storing instructions that when executed by one or more processors cause the one or more processors to:
receive sensor data that includes a location of the object in an environment of the object;
access a location-specific latent map, the location-specific latent map having been learned together with a neural-network-based trajectory predictor during a training phase, wherein the neural-network-based trajectory predictor is deployed in a robot;
input, to the neural-network-based trajectory predictor, the location of the object and the location-specific latent map, the location-specific latent map providing, to the neural-network-based trajectory predictor, a set of location-specific biases regarding the environment of the object; and
output, from the neural-network-based trajectory predictor, a predicted trajectory of the object.

12. The non-transitory computer-readable medium of claim 11 , wherein the set of location-specific biases regarding the environment of the object pertains to both visual and non-visual features of the environment of the object.

13. A method of predicting a trajectory of an object, the method comprising:
receiving sensor data that includes a location of the object in an environment of the object;
accessing a location-specific latent map, the location-specific latent map having been learned together with a neural-network-based trajectory predictor during a training phase, wherein the neural-network-based trajectory predictor is deployed in a robot;
inputting, to the neural-network-based trajectory predictor, the location of the object and the location-specific latent map, the location-specific latent map providing, to the neural-network-based trajectory predictor, a set of location-specific biases regarding the environment of the object; and
outputting, from the neural-network-based trajectory predictor, a predicted trajectory of the object.

14. The method of claim 13 , wherein the robot is an ego vehicle and the object is a road agent external to the ego vehicle.

15. The method of claim 14 , wherein the ego vehicle is an autonomous vehicle.

16. The method of claim 14 , wherein the predicted trajectory of the object includes information regarding roadway geometries inferred by the neural-network-based trajectory predictor.

17. The method of claim 13 , wherein the robot and the object are one and the same thing and the robot is an ego vehicle that is at least partially controlled by a human driver.

18. The method of claim 17 , wherein the predicted trajectory of the object includes information regarding roadway geometries inferred by the neural-network-based trajectory predictor.

19. The method of claim 13 , wherein the robot is deployed inside a building.

20. The method of claim 13 , further comprising encoding the location-specific latent map using a convolutional neural network.

Assignees

• Toyota Res Inst Inc
• Massachusetts Inst Technology

Inventors

• Rosman Guy
• Gilitschenski Igor
• Gupta Arjun
• Karaman Sertac
• Rus Daniela

Classifications

• G06V20/56

  exterior to a vehicle by using sensors mounted on the vehicle · CPC title

• G06V10/82

  using neural networks · CPC title

• G06V10/764

  using classification, e.g. of video objects · CPC title

• G07C5/0808 Primary

  Diagnosing performance data (testing of vehicles G01M17/00; testing of electrical installation on vehicles G01R31/005) · CPC title

• B60W50/0205 Primary

  Diagnosing or detecting failures; Failure detection models · CPC title