Object motion prediction and autonomous vehicle control

Patent metadata

Field	Value
Publication number	US-10656657-B2
Application number	US-201715783005-A
Country	US
Kind code	B2
Filing date	Oct 13, 2017
Priority date	Aug 8, 2017
Publication date	May 19, 2020
Grant date	May 19, 2020

Abstract

Systems and methods for predicting object motion and controlling autonomous vehicles are provided. In one example embodiment, a computer implemented method includes obtaining state data indicative of at least a current or a past state of an object that is within a surrounding environment of an autonomous vehicle. The method includes obtaining data associated with a geographic area in which the object is located. The method includes generating a combined data set associated with the object based at least in part on a fusion of the state data and the data associated with the geographic area in which the object is located. The method includes obtaining data indicative of a machine-learned model. The method includes inputting the combined data set into the machine-learned model. The method includes receiving an output from the machine-learned model. The output can be indicative of a predicted trajectory of the object.

First claim

What is claimed is:

1. A computer-implemented method, comprising:
obtaining, by a computing system comprising one or more computing devices, state data indicative of at least a current or a past state of an object that is within a surrounding environment of an autonomous vehicle;
obtaining, by the computing system, data associated with a geographic area in which the object is located;
generating, by the computing system, a combined data set associated with the object based at least in part on a fusion of the state data and the data associated with the geographic area in which the object is located,
wherein the combined data set comprises image data based at least in part on the fusion of the state data and the data associated with the geographic area in which the object is located, wherein the image data is encoded with one or more features, wherein each of the one or more features is encoded to a different channel of a plurality of channels, wherein the plurality of channels comprises a plurality of color channels, wherein each of the color channels of the plurality of color channels is encoded with a different feature of the one or more features;
obtaining, by the computing system, data indicative of a machine-learned model;
inputting, by the computing system, the combined data set into the machine-learned model, wherein the machine-learned model is configured to receive the one or more features via the plurality of channels; and
receiving, by the computing system, an output from the machine-learned model, wherein the output is indicative of a predicted trajectory of the object.

2. The computer-implemented method of claim 1 , wherein generating, by the computing system, the combined data set associated with the object based at least in part on the fusion of the state data and the data associated with the geographic area in which the object is located comprises:
generating the image data based at least in part on the fusion of the state data and the data associated with the geographic area in which the object is located,
and wherein inputting the combined data set into the machine-learned model comprises inputting the image data into the machine-learned model, wherein the machine-learned model is configured to determine the predicted trajectory of the object based at least in part on the image data.

3. The computer-implemented method of claim 2 , wherein the state data and the data associated with the geographic area in which the object is located are encoded within the image data.

4. The computer-implemented method of claim 2 , wherein the image data comprises a plurality of rasterized images that include the object.

5. The computer-implemented method of claim 4 , wherein inputting the combined data set into the machine-learned model comprises inputting the plurality of rasterized images into the machine-learned model, wherein the machine-learned model is configured to determine the predicted trajectory of the object based at least in part on the plurality of rasterized images.

6. The computer-implemented method of claim 5 , wherein each of the plurality of rasterized images is encoded with the one or more features.

7. The computer-implemented method of claim 1 , wherein the machine-learned model comprises a deep neural network.

8. The computer-implemented method of claim 1 , wherein the data associated with the geographic area in which the object is located comprises at least one of map data associated with the geographic area, sensor data associated with the geographic area, or satellite image data associated with the geographic area.

9. The computer-implemented method of claim 1 , wherein the predicted trajectory of the object comprises a plurality of predicted future locations over time.

10. The computer-implemented method of claim 1 , wherein the output comprises a predicted velocity of the object.

11. The computer-implemented method of claim 1 , further comprising:
controlling, by the computing system, a motion of the autonomous vehicle based at least in part on the output from the machine-learned model.

12. A computing system, comprising:
one or more processors; and
one or more tangible, non-transitory, computer readable media that collectively store instructions that when executed by the one or more processors cause the computing system to perform operations, the operations comprising:
obtaining state data associated with an object that is within a surrounding environment of an autonomous vehicle;
obtaining data associated with a geographic area in which the object is located;
generating image data associated with the object based at least in part on a fusion of the state data associated with the object and the data associated with the geographic area in which the object is located, wherein the image data is encoded with one or more features, wherein each of the one or more features is encoded to a different channel of a plurality of channels, wherein the plurality of channels comprises a plurality of color channels, wherein each of the color channels of the plurality of color channels is encoded with a different feature of the one or more features;
determining a predicted trajectory of the object based at least in part on the image data associated with the object and a machine-learned model; and
planning a motion of the autonomous vehicle based at least in part on the predicted trajectory of the object.

13. The computing system of claim 12 , wherein determining the predicted trajectory of the object based at least in part on the image data associated with the object and the machine-learned model comprises:
obtaining data indicative of the machine-learned model from an accessible memory located onboard the autonomous vehicle;
inputting the image data associated with the object into the machine-learned model; and
receiving an output from the machine-learned model, wherein the output is indicative of the predicted trajectory of the object.

14. The computing system of claim 12 , wherein the machine-learned model is configured to receive each different feature via the plurality of color channels.

15. The computing system of claim 12 , wherein the machine-learned model comprises a convolutional neural network.

16. An autonomous vehicle comprising:
one or more processors; and
one or more tangible, non-transitory, computer readable media that collectively store instructions that when executed by the one or more processors cause the autonomous vehicle to perform operations, the operations comprising:
obtaining state data associated with an object that is within a surrounding environment of the autonomous vehicle;
obtaining data associated with a geographic area in which the object is located;
generating a combined data set associated with the object based at least in part on the state data associated with the object and the data associated with the geographic area in which the object is located, wherein the combined data set comprises a plurality of images that include the object, wherein the plurality of images is indicative of the state data associated with the object and the data associated with the geographic area in which the object is located at various times,
wherein the plurality of images is encoded with one or more features, wherein each of the one or more features is encoded to a different channel of a plurality of channels wherein the plurality of channels comprises a plurality of color channels, wherein each of the color channels of the plurality of color channels is encoded with a different feature of the one or more features; and
determining a predicted

Assignees

• Uatc Llc

Inventors

• Djuric Nemanja
• Radosavljevic Vladan
• Nguyen Thi Duong
• Lin Tsung-Han
• Schneider Jeff

Classifications

• B60W2420/403

  Image sensing, e.g. optical camera · CPC title

• B60W60/00276

  for two or more other traffic participants · CPC title

• G06K9/00369

  Physics · mapped topic

• G05D1/0221

  involving a learning process · CPC title

• G06K9/00805

  Physics · mapped topic