Temporal prediction model for semantic intent understanding

Patent metadata

Field	Value
Publication number	US-10627818-B2
Application number	US-201815939192-A
Country	US
Kind code	B2
Filing date	Mar 28, 2018
Priority date	Mar 28, 2018
Publication date	Apr 21, 2020
Grant date	Apr 21, 2020

Abstract

A temporal prediction model for semantic intent understanding is described. An agent (e.g., a moving object) in an environment can be detected in sensor data collected from sensors on a vehicle. Computing device(s) associated with the vehicle can determine, based partly on the sensor data, attribute(s) of the agent (e.g., classification, position, velocity, etc.), and can generate, based partly on the attribute(s) and a temporal prediction model, semantic intent(s) of the agent (e.g., crossing a road, staying straight, etc.), which can correspond to candidate trajectory(s) of the agent. The candidate trajectory(s) can be associated with weight(s) representing likelihood(s) that the agent will perform respective intent(s). The computing device(s) can use one (or more) of the candidate trajectory(s) to determine a vehicle trajectory along which a vehicle is to drive.

First claim

What is claimed is:

1. A system associated with predicting an intent of an object in an environment proximate an autonomous vehicle, the system comprising:
one or more sensors, wherein the one or more sensors are associated with the autonomous vehicle;
one or more processors; and
one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform actions comprising:
receiving sensor data from the one or more sensors;
detecting, based at least in part on a portion of the sensor data, the object in the environment;
determining, based at least in part on the sensor data, at least one attribute of the object, the at least one attribute comprising a classification distribution of the object indicating individual probabilities that the object is associated with individual classes of a plurality of classes;
generating, based at least in part on the attribute and using a machine learned model, a set of intents;
associating, based at least in part on the attribute, one or more weights with a subset of intents, the subset of intents corresponding to a subset of candidate trajectories;
determining a weight associated with a candidate trajectory of the subset of candidate trajectories meets or exceeds a threshold, the candidate trajectory corresponding to the intent; and
determining a vehicle trajectory along which the autonomous vehicle is to drive based at least in part on the intent of the object.

2. The system as claim 1 recites, wherein the one or more sensors comprise at least a light detection and ranging (LIDAR) sensor, a radio detection and ranging (RADAR) sensor, a sound navigation and ranging (SONAR) sensor, a location sensor, an inertial sensor, a camera, a microphone, or an environment sensor.

3. The system as claim 1 recites, the actions further comprising accessing map data associated with a map of the environment, wherein the at least one attribute further comprises at least a portion of the map data.

4. The system as claim 1 recites, wherein the plurality of classes comprise at least a vehicle class, a pedestrian class, or a bicycle class.

5. The system as claim 1 recites, wherein the intent of the object comprises:
staying stationary;
moving in a straight line;
moving along a road;
changing lanes;
following a sidewalk associated with the road;
crossing a crosswalk associated with the road; or
jaywalking relative to the crosswalk and across the road.

6. A method comprising:
receiving sensor data from one or more sensors;
determining, based at least in part on a portion of the sensor data, an object in an environment;
determining, based at least in part on the sensor data, one or more attributes of the object, wherein an attribute of the one or more attributes comprises a probability distribution indicating one or more probabilities that the object is associated with one or more classes of objects;
generating, based at least in part on the one or more attributes and a prediction model, a set of intents corresponding to a set of candidate trajectories;
determining, based at least in part on the one or more attributes, a candidate trajectory of the set of candidate trajectories, the candidate trajectory corresponding to an intent of the object; and
determining a vehicle trajectory along which a vehicle is to drive based at least in part on the intent of the object.

7. The method as claim 6 recites, wherein the one or more attributes further comprise:
a position of the object;
an orientation of the object;
a heading of the object;
a velocity of the object;
an acceleration of the object;
an extent of the object; or
a semantic attribute associated with the object.

8. The method as claim 6 recites, wherein the one or more classes include at least one of a vehicle class, a pedestrian class, or a bicycle class.

9. The method as claim 6 recites, further comprising accessing map data associated with a map of the environment, wherein the one or more attributes further comprise at least a portion of the map data.

10. The method as claim 6 recites, further comprising associating, based at least in part on the one or more attributes, one or more weights with the set of candidate trajectories.

11. The method as claim 10 recites, wherein associating the one or more weights with the set of candidate trajectories is based at least in part on a machine learned model.

12. The method as claim 6 recites, wherein the set of intents of the object comprises:
staying stationary;
moving in a straight line;
moving along a road;
changing lanes;
following a sidewalk associated with the road;
crossing a crosswalk associated with the road; or
jaywalking relative to the crosswalk and across the road.

13. The method as claim 6 recites, further comprising refining a subset of the set of candidate trajectories based at least in part on determining one or more motion profiles.

14. The method as claim 6 recites, further comprising, associating, based at least in part on the one or more attributes, one or more weights with the set of candidate trajectories, wherein the weights are smoothed according to a probabilistic temporal smoothing model.

15. The method as claim 6 recites, wherein the vehicle is an autonomous vehicle, and the method further comprises executing, by a computing device associated with the autonomous vehicle, the vehicle trajectory to cause the autonomous vehicle to drive along the vehicle trajectory.

16. A non-transitory computer-readable medium having a set of instructions that, when executed, cause one or more processors to perform operations comprising:
receiving, from one or more sensors, sensor data associated with an environment;
determining, based at least in part on the sensor data, an object in the environment;
determining, based at least in part on the sensor data, a plurality of attributes of the object or the environment, wherein an attribute of the plurality of attributes comprises a distribution of confidence scores indicating levels of confidence that the object is associated with individual classes of one or more classes of objects;
determining, based at least in part on the plurality of attributes and a machine learned model, a set of intents of the object corresponding to a set of candidate trajectories;
associating a set of weights with the set of candidate trajectories;
selecting, based at least in part on the set of weights associated with the set of candidate trajectories, one or more candidate trajectories of the set of candidate trajectories; and
determining, based at least in part on the one or more candidate trajectories, a vehicle trajectory along which a vehicle is to drive.

17. The non-transitory computer-readable medium as claim 16 recites, wherein the plurality of attributes further comprise at least one of:
a position of the object;
an orientation of the object;
a heading of the object;
a velocity of the object;
an acceleration of the object;
an extent of the object;
a semantic attribute associated with the object; or
a semantic attribute associated with the environment.

18. The non-transitory computer-readable medium as claim 16 recites, the operations further comprising accessing map data associated with a map of the environment, wherein the plurality of attributes further comprises the map data and the map data comprises at least one of semantic features, static structures, or road topologies.

19. The non-trans

Assignees

• Zoox Inc

Inventors

• Sapp Benjamin John
• Wang Yilun

Classifications

• B60W2554/4041

  Position · CPC title

• B60W60/00274 Primary

  considering possible movement changes · CPC title

• B60W60/00276 Primary

  for two or more other traffic participants · CPC title

• B60W2554/4026

  Cycles · CPC title

• B60W2552/45

  Pedestrian sidewalk · CPC title