Prediction on top-down scenes based on object motion

Patent metadata

Field	Value
Publication number	US-2021192748-A1
Application number	US-201916719780-A
Country	US
Kind code	A1
Filing date	Dec 18, 2019
Priority date	Dec 18, 2019
Publication date	Jun 24, 2021
Grant date	—

Abstract

Techniques for determining predictions on a top-down representation of an environment based on object movement are discussed herein. Sensors of a first vehicle (such as an autonomous vehicle) may capture sensor data of an environment, which may include object(s) separate from the first vehicle (e.g., a vehicle, a pedestrian, a bicycle). A multi-channel image representing a top-down view of the object(s) and the environment may be generated based in part on the sensor data. Environmental data (object extents, velocities, lane positions, crosswalks, etc.) may also be encoded in the image. Multiple images may be generated representing the environment over time and input into a prediction system configured to output a trajectory template (e.g., general intent for future movement) and a predicted trajectory (e.g., more accurate predicted movement) associated with each object. The prediction system may include a machine learned model configured to output the trajectory template(s) and the predicted trajector(ies).

First claim

What is claimed is:

1 . A method comprising:
receiving sensor data of an environment captured by a sensor of an autonomous vehicle; generating, based at least in part on the sensor data, a multi-channel image representing a top-down view of the environment, the multi-channel image representing a bounding box associated with an object in the environment inputting the multi-channel image into a machine learned model; receiving, from the machine learned model, a trajectory template indicative of a class of motion associated with the object and a predicted trajectory associated with the object; and controlling the autonomous vehicle based at least in part on at least one of the trajectory template or the predicted trajectory.

2 . The method of claim 1 , wherein:
the object is a first object; the multi-channel image comprises data associated with a second object; the trajectory template is a first trajectory template; and the first trajectory template and a second trajectory template are based at least in part an interaction between the first object and the second object.

3 . The method of claim 1 , wherein the machine learned model comprises:
a first neural network to output a feature map determined based at least in part on a history of the object in the environment; a second neural network to output the trajectory template based at least in part on a feature vector associated with the feature map; and a third neural network to output the predicted trajectory based at least in part on the trajectory template and the feature vector.

4 . The method of claim 3 , wherein:
the second neural network is further configured to output a heat map associated with the object; and the autonomous vehicle is further controlled based at least in part on the heat map.

5 . The method of claim 3 , wherein the third neural network outputs the predicted trajectory based at least in part on a first classification of trajectory templates and wherein the machine learned model further comprises:
a fourth neural network to output the predicted trajectory based at least in part on a second classification of the trajectory templates and the feature vector, the method further comprising inputting the trajectory template into one or more of the third neural network or the fourth neural network based at least in part on a classification associated with the trajectory template.

6 . A system comprising:
one or more processors; and one or more computer-readable media storing instructions executable by the one or more processors, wherein the instructions, when executed, cause the system to perform operations comprising:
receiving data representing an object in an environment;
generating, based at least in part on the data, an image representing a top-down view of the environment, the image representing the object and motion information associated with the object;
inputting the image into a machine learned model; and
receiving, from the machine learned model, a trajectory template and a predicted trajectory associated with the object, the trajectory template indicative of a class of motion associated with the object.

7 . The system of claim 6 , wherein the predicted trajectory is represented as a heat map comprising prediction probabilities of possible locations associated with the object.

8 . The system of claim 6 , wherein the machine learned model comprises:
a first neural network to output a feature map; a second neural network to output the trajectory template based at least in part on a portion of the feature map associated with the object; and a third neural network to output the predicted trajectory based at least in part on the trajectory template.

9 . The system of claim 8 , wherein the third neural network outputs the predicted trajectory based at least in part on a plurality of trajectory templates.

10 . The system of claim 8 , wherein the third neural network outputs the predicted trajectory based at least in part on a first classification of the trajectory template, the machine learned model further comprising:
a fourth neural network to output the predicted trajectory based at least in part on a second classification of the trajectory template.

11 . The system of claim 8 , wherein:
the object data comprises data associated with two or more objects in the environment; and the feature map comprises joint history data of the two or more objects determined in the environment.

12 . The system of claim 6 , wherein:
the image further comprises one or more channels comprising additional object information of additional objects in the environment; and the image is one of a plurality of images associated with one or more previous times prior to a current time.

13 . The system of claim 12 , wherein the image represents an interaction between the object and the additional objects and wherein the predicted trajectory and at least one other predicted trajectory associated with an additional object are based at least in part the interaction.

14 . The system of claim 6 , the operations further comprising causing the system to control a vehicle based at least in part on at least one of the trajectory template or the predicted trajectory associated with the object.

15 . The system of claim 6 , wherein the image is a multi-channel image comprising at least one of:
semantic data associated with object; velocity data associated with the object; acceleration data associated with the object; scenario data associated with the environment; a road network associated with the environment; or vehicle data associated with a vehicle in the environment.

16 . The system of claim 6 , wherein the machine learned model is trained to:
determine the trajectory template based at least in part on a clustering algorithm; and determine the predicted trajectory based at least in part on a regression algorithm.

17 . One or more non-transitory computer-readable media storing instructions that, when executed, cause one or more processors to perform operations comprising:
receiving data representing an object in an environment; generating, based at least in part on the data, an image representing at least a top-down view of the environment; inputting the image into a machine learned model; and receiving, from the machine learned model, a trajectory template and a predicted trajectory associated with the object, the trajectory template indicative of a class of motion associated with the object.

18 . The one or more non-transitory computer-readable media of claim 17 , wherein the image is a multi-channel image comprising at least one of:
semantic data associated with object; velocity data associated with the object; acceleration data associated with the object; scenario data associated with the environment; a road network associated with the environment; or vehicle data associated with a vehicle in the environment.

19 . The one or more non-transitory computer-readable media of claim 17 , wherein the machine learned model comprises:
a first neural network to output the trajectory template; and at least one second neural network to output the predicted trajectory based at least in part on the output trajectory template.

20 . The one or more non-transitory computer-readable media of claim 17 , wherein the operations further comprise:
controlling a vehicle based at least in part on at least one of the trajectory template or the predicted trajectory associated with th

Assignees

• Zoox Inc

Inventors

• Morales Morales Andres Guillermo
• Kobilarov Marin
• Garimella Gowtham
• Wang Kai Zhenyu

Classifications

• G06T7/277 Primary

  involving stochastic approaches, e.g. using Kalman filters · CPC title

• G06V20/588

  Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road · CPC title

• G06V20/582

  of traffic signs · CPC title

• G06V20/584

  of vehicle lights or traffic lights · CPC title

• G06V30/19173

  Classification techniques · CPC title