Conditional trajectory determination by a machine learned model

Patent metadata

Field	Value
Publication number	US-12311972-B2
Application number	US-202217855671-A
Country	US
Kind code	B2
Filing date	Jun 30, 2022
Priority date	Jun 30, 2022
Publication date	May 27, 2025
Grant date	May 27, 2025

Abstract

Techniques for predicting an object trajectory or scene information are described herein. For example, the techniques may include inputting tokens representing discrete behavior into a machine learned model. The machine learned model may output a sequence of tokens that is usable by another machine learned model to generate an object trajectory (e.g., position data, velocity data, acceleration data, etc.) for one or more objects in the environment. The object trajectory can be sent to a vehicle computing device for consideration during vehicle planning, which may include simulation.

First claim

What is claimed is:

1. A method comprising:
receiving sensor data from a sensor associated with a vehicle;
receiving, by a first machine learned model, a set of tokens from a codebook, wherein:
at least one token in the codebook represents a potential characteristic of an object in an environment,
the potential characteristic of the object represents a state or an action,
the codebook comprises the set of tokens for processing by a second machine learned model,
the first machine learned model implements an autoregressive algorithm to sample the set of tokens from the codebook,
the first machine learned model comprises one or more self-attention layers configured to determine scores for at least some of the set of tokens from the codebook, a first score indicating a dependency between a first token and a second token and a second score indicating a dependency between a third token and one of: the first token, the second token, or a fourth token, and
determining the set of tokens is further based at least in part on the one or more self-attention layers determining the scores,

inputting the set of tokens into the second machine learned model;
determining, by the second machine learned model and based at least in part on the set of tokens, an object trajectory for the object to follow in the environment; and
controlling the vehicle in the environment based at least in part on the object trajectory.

2. The method of claim 1 , wherein:
an order of the set of tokens is based at least in part on the autoregressive algorithm;
the first machine learned model is a transformer model, and
the second machine learned model comprises at least one of a Generative Adversarial Network (GAN), a Graph Neural Network (GNN), a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), or another transformer model.

3. The method of claim 1 , wherein:
the first machine learned model comprises a transformer model, and
the transformer model is trained based at least in part on a set of conditions, at least one condition of the set of conditions comprising a previous action, a previous position, or a previous acceleration of the object.

4. The method of claim 1 , wherein:
the set of tokens in the codebook represent discrete latent variables.

5. The method of claim 1 , wherein:
the first token of the set of tokens represents the action for the vehicle in the environment to take at a future time, and
the second token of the set of tokens represents a feature of the object in the environment.

6. The method of claim 5 , wherein:
the first token represents one of: a yield action, a drive straight action, a left turn action, a right turn action, a brake action, an acceleration action, a steering action, or a lane change action, and
the second token represents a position, a heading, or an acceleration of the object.

7. The method of claim 1 , wherein:
the codebook comprises a fixed number of tokens, and determining the set of tokens is further based at least in part on the fixed number of tokens in the codebook.

8. The method of claim 1 , wherein the set of tokens represent discrete latent variables, and further comprising:
mapping the discrete latent variables associated with the set of tokens to continuous variables associated with feature vectors,
wherein inputting the set of tokens into the second machine learned model comprises inputting the continuous variables associated with the feature vectors, and
determining, by the second machine learned model, the object trajectory is based at least in part on the continuous variables associated with the feature vectors.

9. The method of claim 1 , wherein:
the object is a first object,
another token in the codebook represents a potential action of a second object in the environment, and
the set of tokens represents a potential interaction between the second object and the first object.

10. The method of claim 1 , wherein:
the environment is a simulated environment,
another token in the codebook represents a feature of the simulated environment, and further comprising:
determining, by the second machine learned model and based at least in part on the set of tokens, scene data for testing or verifying a scenario in the simulated environment.

11. One or more non-transitory computer-readable media storing instructions that, when executed, cause one or more processors to perform actions comprising:
receiving sensor data from a sensor associated with a vehicle;
receiving, by a first machine learned model, a set of tokens from a codebook, wherein:
at least one token in the codebook represents a potential characteristic of an object in an environment,
the potential characteristic of the object represents a state or an action,
the codebook comprises the set of tokens for processing by a second machine learned model,
the first machine learned model implements an autoregressive algorithm to sample the set of tokens from the codebook,
the first machine learned model comprises one or more self-attention layers configured to determine scores for at least some of the set of tokens from the codebook, a first score indicating a dependency between a first token and a second token and a second score indicating a dependency between a third token and one of: the first token, the second token, or a fourth token, and
determining the set of tokens is further based at least in part on the one or more self-attention layers determining the scores,

inputting the set of tokens into the second machine learned model;
determining, by the second machine learned model and based at least in part on the set of tokens, an object trajectory for the object to follow in the environment; and
controlling the vehicle in the environment based at least in part on the object trajectory.

12. The one or more non-transitory computer-readable media of claim 11 , wherein:
an order of the set of tokens is based at least in part on the autoregressive algorithm;
the first machine learned model is a transformer model, and
the second machine learned model comprises at least one of a Generative Adversarial Network (GAN), a Graph Neural Network (GNN), a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), or another transformer model.

13. The one or more non-transitory computer-readable media of claim 11 , wherein:
the first machine learned model comprises a transformer model, and
the transformer model is trained based at least in part on a set of conditions, at least one condition of the set of conditions comprising a previous action, a previous position, or a previous acceleration of the object.

14. The one or more non-transitory computer-readable media of claim 11 , wherein:
the set of tokens in the codebook represent discrete latent variables.

15. A system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the one or more processors to perform actions comprising:
receiving sensor data from a sensor associated with a vehicle;
receiving, by a first machine learned model, a set of tokens from a codebook, wherein:
at least one token in the codebook represents a potential characteristic of an object in an environment,
the potential characteristic of the object represents a state or an action,
the codebook comprises the set of tokens for processing by a second machine learned model,
the first machine learned model implements an autoregressive algorithm to sample the set of tokens from the codeboo

Assignees

• Zoox Inc

Inventors

• Pronovost Ethan Miller

Classifications

• B60W40/04

  Traffic conditions · CPC title

• B60W2554/4046

  Behavior, e.g. aggressive or erratic · CPC title

• B60W2554/4045

  Intention, e.g. lane change or imminent movement · CPC title

• G06N20/00

  Machine learning · CPC title

• B60W50/0097

  Predicting future conditions · CPC title