Systems and methods for generating motion forecast data for actors with respect to an autonomous vehicle and training a machine learned model for the same

Patent metadata

Field	Value
Publication number	US-12008454-B2
Application number	US-202318186718-A
Country	US
Kind code	B2
Filing date	Mar 20, 2023
Priority date	Jul 8, 2019
Publication date	Jun 11, 2024
Grant date	Jun 11, 2024

Abstract

Systems and methods for generating motion forecast data for actors with respect to an autonomous vehicle and training a machine learned model for the same are disclosed. The computing system can include an object detection model and a graph neural network including a plurality of nodes and a plurality of edges. The computing system can be configured to input sensor data into the object detection model; receive object detection data describing the location of the plurality of the actors relative to the autonomous vehicle as an output of the object detection model; input the object detection data into the graph neural network; iteratively update a plurality of node states respectively associated with the plurality of nodes; and receive, as an output of the graph neural network, the motion forecast data with respect to the plurality of actors.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving, from one or more sensors, sensor data associated with an environment of an autonomous vehicle;
generating, using an object detection model, object detection data based on the sensor data, the object detection data describing a location of an actor relative to the autonomous vehicle;
generating, using a graph neural network, motion forecast data associated with the actor, wherein the graph neural network comprises a plurality of nodes and a plurality of edges, the plurality of nodes indicative of a plurality of actors including the actor and the edges indicative of relationships among the plurality of actors,
wherein the graph neural network is configured to update at least one node state of a first node by passing at least one message between at least two nodes comprising the first node; and
determining a motion plan for the autonomous vehicle based on the motion forecast data; and
controlling the autonomous vehicle based on the motion plan.

2. The computer-implemented method claim 1 , comprising:
obtaining map data associated with the environment of the autonomous vehicle;
concatenating, using the object detection model, the map data and the sensor data; and
generating, using the object detection model, the object detection data based on the map data and the sensor data.

3. The computer-implemented method of claim 1 , wherein the graph neural network is indicative of at least one of: (i) a partially connected graph neural network, or (ii) a fully connected graph neural network.

4. The computer-implemented method of claim 1 , wherein the at least one node state is indicative of at least one of: (i) a future trajectory of a respective actor of the plurality of actors or (ii) a behavior of the respective actor.

5. The computer-implemented method of claim 1 , wherein passing the at least one message between the at least two nodes is indicative of a potential interaction among the actors respectively associated with the at least two nodes.

6. The computer-implemented method of claim 5 , wherein the at least one message is indicative of a spatial relationship.

7. The computer-implemented method of claim 1 , wherein the sensor data comprises three-dimensional LiDAR data.

8. A vehicle computing system comprising:
one or more processors; and
one or more memory resources storing instructions executable by the one or more processors, to cause the one or more processors to:
receive, from one or more sensors, sensor data associated with an environment of an autonomous vehicle;
generate, using an object detection model, object detection data based on the sensor data, the object detection data describing a location of an actor relative to the autonomous vehicle;
generate, using a graph neural network, motion forecast data associated with the actor, wherein the graph neural network comprises a plurality of nodes and a plurality of edges, the plurality of nodes indicative of a plurality of actors including the actor and the edges indicative of relationships among the plurality of actors,
wherein the graph neural network is configured to update at least one node state of a first node by passing at least one message between at least two nodes comprising the first node; and
determine a motion plan for the autonomous vehicle based on the motion forecast data; and
control the autonomous vehicle based on the motion plan.

9. The vehicle computing system of claim 8 , wherein the instructions cause the one or more processors to:
obtain map data associated with the environment of the autonomous vehicle;
concatenate, using the object detection model, the map data and the sensor data; and
generate, using the object detection model, the object detection data based on the map data and the sensor data.

10. The vehicle computing system of claim 8 , wherein t graph neural network is indicative of at least one of: (i) a partially connected graph neural network, or (ii) a fully connected graph neural network.

11. The vehicle computing system of claim 8 , wherein the at least one node state is indicative of at least one of: (i) a future trajectory of a respective actor of the plurality of actors or (ii) a behavior of the respective actor.

12. The vehicle computing system of claim 8 , wherein passing the at least one message between the at least two nodes is indicative of a potential interaction among the actors respectively associated with the at least two nodes.

13. The vehicle computing system of claim 12 , wherein the at least one message is indicative of a spatial relationship.

14. The vehicle computing system of claim 8 , wherein the sensor data comprises three-dimensional LiDAR data.

15. A non-transitory computer-readable medium storing instructions executable by one or more processors of an autonomous vehicle computing system, to cause the one or more processors to:
receive, from one or more sensors, sensor data associated with an environment of an autonomous vehicle;
generate, using an object detection model, object detection data based on the sensor data, the object detection data describing a location of an actor relative to the autonomous vehicle;
generate, using a graph neural network, motion forecast data associated with the actor, wherein the graph neural network comprises a plurality of nodes and a plurality of edges, the plurality of nodes indicative of a plurality of actors including the actor and the edges indicative of relationships among the plurality of actors,
wherein the graph neural network is configured to update at least one node state of a first node by passing at least one message between at least two nodes comprising the first node; and
determine a motion plan for the autonomous vehicle based on the motion forecast data; and
control the autonomous vehicle based on the motion plan.

16. The non-transitory computer-readable medium of claim 15 , wherein the instructions cause the one or more processors to:
obtain map data associated with the environment of the autonomous vehicle;
concatenate, using the object detection model, the map data and the sensor data; and
generate, using the object detection model, the object detection data based on the map data and the sensor data.

17. The non-transitory computer-readable medium of claim 15 , wherein the graph neural network is indicative of at least one of (i) a partially connected graph neural network, or (ii) a fully connected graph neural network.

18. The non-transitory computer-readable medium of claim 15 , wherein the at least one node state is indicative of at least one of: (i) a future trajectory of a respective actor of the plurality of actors or (ii) a behavior of the respective actor.

19. The non-transitory computer-readable medium of claim 15 , wherein passing the at least one message between the at least two nodes is indicative of a potential interaction among the actors respectively associated with the at least two nodes.

20. The non-transitory computer-readable medium of claim 19 , wherein the at least one message is indicative of a spatial relationship.

Assignees

• Uatc Llc

Inventors

• Urtasun Raquel
• Liao Renjie
• Casas Sergio
• Gulino Cole Christian

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N3/045

  Combinations of networks · CPC title