Machine-learned model for detecting object relevance to vehicle operation planning

Patent metadata

Field	Value
Publication number	US-12415549-B1
Application number	US-202318132289-A
Country	US
Kind code	B1
Filing date	Apr 7, 2023
Priority date	Apr 7, 2023
Publication date	Sep 16, 2025
Grant date	Sep 16, 2025

Abstract

A machine-learned architecture for determining whether an object is relevant to a vehicle's action planning may comprise a convolutional neural network, graph neural network, and/or multi-layer perceptron that may determine a relevance score associated with an object that indicates indicating whether an object is likely to impact operation(s) of a vehicle. In some examples, the machine-learned architecture may use scene information and/or an object track to determine the relevance score.

First claim

What is claimed is:

1. A system comprising:
one or more processors; and
a non-transitory memory storing processor-executable instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
receiving sensor data;
determining, based at least in part on the sensor data, a top-down representation of an environment;
determining, based at least in part on the sensor data, a track associated with an object, the track representing a state of the object at multiple timesteps;
determining, by a first machine-learned model and based at least in part on the top-down representation, a first intermediate output;
determining, by a second machine-learned model and based at least in part on the track, a second intermediate output;
determining, by a third machine-learned model and based at least in part on the first intermediate output and the second intermediate output, a relevance score associated with the object indicating an extent to which an action associated with the object is associated with a vehicle; and
controlling the vehicle based at least in part on the relevance score.

2. The system of claim 1 , wherein the first machine-learned model comprises a convolutional neural network, the second machine-learned model comprises a graph neural network, and the third machine-learned model comprises a multi-layer perceptron.

3. The system of claim 1 , wherein:
the second intermediate output comprises a graph feature; and
the operations further comprise determining a first portion of the graph feature associated with the object and associating the first portion with a second portion of the first intermediate output.

4. The system of claim 1 , wherein:
the operations further comprise determining, by a fourth machine-learned model and based at least in part on the first intermediate output and the second intermediate output, an estimated action score associated with a future action of the object; and
the third machine-learned model determines the relevance score further based at least in part on the estimated action score.

5. The system of claim 1 , wherein the object is a first object and controlling the vehicle comprises:
determining, by a planning component and based at least in part on first data associated with a second object, a trajectory for controlling the vehicle; and
determining that the relevance score associated with the first object is less than or equal to a threshold score,
wherein determining the trajectory is independent of second data associated with the first object based at least in part on determining that the relevance score is less than or equal to the threshold score.

6. The system of claim 1 , wherein:
the object is a first object, the vehicle is a first vehicle, and the environment is a first environment;
the first machine-learned model, the second machine-learned model, and the third machine-learned model are trained using training data determined from log data indicating movement of a second object in a second environment relative to a second vehicle or the first vehicle; and
generating the training data comprises determining, based at least in part on a rule set and the log data, a training score associated with the second object.

7. One or more non-transitory computer-readable media storing processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving sensor data;
determining, based at least in part on the sensor data, a top-down representation of an environment;
determining, based at least in part on the sensor data, a track associated with an object, the track representing a state of the object at multiple timesteps;
determining, by a first machine-learned model and based at least in part on the top-down representation, a first intermediate output;
determining, by a second machine-learned model and based at least in part on the track, a second intermediate output;
determining, by a third machine-learned model and based at least in part on the first intermediate output and the second intermediate output, a relevance score associated with the object indicating an extent to which an action associated with the object is associated with a vehicle; and
controlling the vehicle based at least in part on the relevance score.

8. The one or more non-transitory computer-readable media of claim 7 , wherein the first machine-learned model comprises a convolutional neural network, the second machine-learned model comprises a graph neural network, and the third machine-learned model comprises a multi-layer perceptron.

9. The one or more non-transitory computer-readable media of claim 7 , wherein:
the second intermediate output comprises a graph feature; and
the operations further comprise determining a first portion of the graph feature associated with the object and associating the first portion with a second portion of the first intermediate output.

10. The one or more non-transitory computer-readable media of claim 7 , wherein:
the operations further comprise determining, by a fourth machine-learned model and based at least in part on the first intermediate output and the second intermediate output, an estimated action score associated with a future action of the object; and
the third machine-learned model determines the relevance score further based at least in part on the estimated action score.

11. The one or more non-transitory computer-readable media of claim 10 , wherein:
the operations further comprise determining, based at least in part on a future action of the object and the estimated action score, a future scenario; and
controlling the vehicle is based at least in part on the future scenario.

12. The one or more non-transitory computer-readable media of claim 7 , wherein the object is a first object and controlling the vehicle comprises:
determining, by a planning component and based at least in part on first data associated with a second object, a trajectory for controlling the vehicle; and
determining that the relevance score associated with the first object is less than or equal to a threshold score,
wherein determining the trajectory is independent of second data associated with the first object based at least in part on determining that the relevance score is less than or equal to the threshold score.

13. The one or more non-transitory computer-readable media of claim 7 , wherein the third machine-learned model determines the relevance score further based at least in part on a graph representation indicating at least one of a lane location, a lane dimension, a junction location, a junction dimension, or an annotation.

14. The one or more non-transitory computer-readable media of claim 7 , wherein:
the object is a first object, the vehicle is a first vehicle, and the environment is a first environment;
the first machine-learned model, the second machine-learned model, and the third machine-learned model are trained using training data determined from log data indicating movement of a second object in a second environment relative to a second vehicle; and
generating the training data comprises determining, based at least in part on a rule set and the log data, a training score associated with the second object.

15. The one or more non-transitory computer-readable media of claim 14 , wherein the rule set comprises a scoring function that is based at least in part on at least one of:
determining, based at least in part on the log data, an interaction between the second object and the second vehicle by:

Assignees

• Zoox Inc

Inventors

• Parikh Samir
• Garimella Gowtham
• Zhang Linjun
• Dai Chunlei
• Emam Yousef Ali
• Wang Kai Zhenyu
• Wang Woodrow Zhouyuan
• Mattinson Benjamin Isaac
• Ren Xiaobo

Classifications

• B60W2530/203

  Presence of trailer · CPC title

• G06N3/08

  Learning methods · CPC title

• G06N3/045

  Combinations of networks · CPC title

• B60W2554/404

  Characteristics · CPC title

• B60W2554/4045

  Intention, e.g. lane change or imminent movement · CPC title