Systems and methods for generating motion forecast data for actors with respect to an autonomous vehicle and training a machine learned model for the same

Patent metadata

Field	Value
Publication number	US-12430534-B2
Application number	US-202418656150-A
Country	US
Kind code	B2
Filing date	May 6, 2024
Priority date	Jul 8, 2019
Publication date	Sep 30, 2025
Grant date	Sep 30, 2025

Abstract

Systems and methods for generating motion forecast data for actors with respect to an autonomous vehicle and training a machine learned model for the same are disclosed. The computing system can include an object detection model and a graph neural network including a plurality of nodes and a plurality of edges. The computing system can be configured to input sensor data into the object detection model; receive object detection data describing the location of the plurality of the actors relative to the autonomous vehicle as an output of the object detection model; input the object detection data into the graph neural network; iteratively update a plurality of node states respectively associated with the plurality of nodes; and receive, as an output of the graph neural network, the motion forecast data with respect to the plurality of actors.

First claim

What is claimed is:

1. A computer-implemented method comprising:
generating, using an object detection model, object detection data describing a location of an actor within an environment of an autonomous vehicle;
generating, using a graph neural network, motion forecast data associated with the actor, wherein:
the graph neural network comprises a plurality of nodes, wherein a first node of the plurality of nodes corresponds to the actor, and
the graph neural network is configured to model anticipated interactions between the actor and other actors in the environment by passing one or more messages among the plurality of nodes to update at least one node state of the first node, wherein the anticipated interactions comprise an interaction between the actor and at least one other actor which alters a trajectory of the actor or the at least one other actor in the environment;
determining a motion plan for the autonomous vehicle based on the motion forecast data; and
controlling the autonomous vehicle based on the motion plan.

2. The computer-implemented method of claim 1 , further comprising:
inputting at least one of: (i) sensor data or (ii) map data into the object detection model.

3. The computer-implemented method of claim 2 , wherein the sensor data is captured by a remote computing system.

4. The computer-implemented method of claim 2 , wherein the sensor data and the map data are concatenated prior to being input into the object detection model.

5. The computer-implemented method of claim 1 , wherein the object detection data comprises a region of interest within the environment, the region of interest associated with the actor.

6. The computer-implemented method of claim 1 , wherein at least a portion of the plurality of nodes correspond to the other actors in the environment.

7. The computer-implemented method of claim 1 , wherein the anticipated interactions comprise an interaction within a threshold distance between the actor and at least one other actor in the environment.

8. A computing system comprising:
one or more processors, and
one or more memory resources storing instructions executable by the one or more processors to cause the one or more processors to:
generate, using an object detection model, object detection data describing a location of an actor within an environment of an autonomous vehicle;
generate, using a graph neural network, motion forecast data associated with the actor, wherein:
the graph neural network comprises a plurality of nodes, wherein a first node of the plurality of nodes corresponds to the actor, and
the graph neural network is configured to model anticipated interactions between the actor and other actors in the environment by passing one or more messages among the plurality of nodes to update at least one node state of the first node, wherein the anticipated interactions comprise an interaction between the actor and at least one other actor which alters a trajectory of the actor or the at least one other actor in the environment;
determine a motion plan for the autonomous vehicle based on the motion forecast data; and
control the autonomous vehicle based on the motion plan.

9. The computing system of claim 8 , wherein the one or more processors:
input at least one of: (i) sensor data or (ii) map data into the object detection model.

10. The computing system of claim 9 , wherein the sensor data is captured by a remote computing system.

11. The computing system of claim 9 , wherein the sensor data and the map data are concatenated prior to being input into the object detection model.

12. The computing system of claim 8 , wherein the object detection data comprises a region of interest within the environment, the region of interest associated with the actor.

13. The computing system of claim 8 , wherein at least a portion of the plurality of nodes correspond to the other actors in the environment.

14. The computing system of claim 8 , wherein the anticipated interactions comprise an interaction within a threshold distance between the actor and at least one other actor in the environment.

15. A non-transitory computer-readable media storing instructions executable by one or more processors of an autonomous vehicle computing system to cause the one or more processors to:
generate, using an object detection model, object detection data describing a location of an actor within an environment of an autonomous vehicle;
generate, using a graph neural network, motion forecast data associated with the actor, wherein:
the graph neural network comprises a plurality of nodes, wherein a first node of the plurality of nodes corresponds to the actor, and
the graph neural network is configured to model anticipated interactions between the actor and other actors in the environment by passing one or more messages among the plurality of nodes to update at least one node state of the first node, wherein the anticipated interactions comprise an interaction between the actor and at least one other actor which alters a trajectory of the actor or the at least one other actor in the environment;
determine a motion plan for the autonomous vehicle based on the motion forecast data; and
control the autonomous vehicle based on the motion plan.

16. The non-transitory computer-readable media of claim 15 , wherein the one or more processors:
input at least one of: (i) sensor data or (ii) map data into the object detection model.

17. The non-transitory computer-readable media of claim 16 , wherein the sensor data is captured by a remote computing system.

18. The non-transitory computer-readable media of claim 16 , wherein the sensor data and the map data are concatenated prior to being input into the object detection model.

19. The non-transitory computer-readable media of claim 15 , wherein the object detection data comprises a region of interest within the environment, the region of interest associated with the actor.

20. The non-transitory computer-readable media of claim 15 , wherein at least a portion of the plurality of nodes correspond to the other actors in the environment.

Assignees

• Uatc Llc
• Aurora Operations Inc

Inventors

• Urtasun Raquel
• Liao Renjie
• Casas Sergio
• Gulino Cole Christian

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/06

  Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons · CPC title

• G06N7/046

  Implementation by means of a neural network (neural networks using fuzzy logic G06N3/043) · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title