Predicting three-dimensional features for autonomous driving

Patent metadata

Field	Value
Publication number	US-12014553-B2
Application number	US-202117450914-A
Country	US
Kind code	B2
Filing date	Oct 14, 2021
Priority date	Feb 1, 2019
Publication date	Jun 18, 2024
Grant date	Jun 18, 2024

Abstract

A processor coupled to memory is configured to receive image data based on an image captured by a camera of a vehicle. The image data is used as a basis of an input to a trained machine learning model trained to predict a three-dimensional trajectory of a machine learning feature. The three-dimensional trajectory of the machine learning feature is provided for automatically controlling the vehicle.

First claim

What is claimed is:

1. A system, comprising:
non-transitory computer storage media storing instructions that when executed by one or more processors, cause the one or more processors to:
obtain sensor data via one or more sensors of a vehicle;
determine, based on a machine learning model, a three-dimensional feature associated with the sensor data,
wherein the machine learning model is trained using a training dataset comprising a determined ground truth and corresponding sensor data captured within a period of time, the corresponding sensor data comprising a plurality of time series elements,
wherein the machine learning model is trained to output the determined ground truth based on an input of at least a portion of the corresponding sensor data comprising a particular time series element of the plurality of time series elements, and
wherein the determined ground truth is indicative of a three-dimensional feature associated with the corresponding sensor data; and

adjust operation of the vehicle based on the three-dimensional feature.

2. The system of claim 1 , wherein the three-dimensional feature is a three-dimensional trajectory of a real-world feature.

3. The system of claim 2 , wherein the real-world feature is a vehicle lane line.

4. The system of claim 2 , wherein the real-world feature is a different vehicle.

5. The system of claim 1 , wherein to adjust operation of the vehicle the instructions cause the one or more processors to cause adjustment of a speed or steering of the vehicle.

6. The system of claim 1 , wherein the three-dimensional feature is determined based on the sensor data and odometry data associated with the vehicle.

7. The system of claim 1 , wherein the ground truth is determined via selecting portions of each of the time series elements which are associated with a highest certainty in depicting respective portions of a real-world feature.

8. The system of claim 7 , wherein the real-world feature is a vehicle lane line and wherein at least one unselected portion occludes the vehicle lane line.

9. The system of claim 8 , wherein the ground truth is determined based on selected portions and elevation information associated with the vehicle lane line.

10. A method implemented by a system of one or more processors, the method comprising:
obtaining sensor data via one or more sensors of a vehicle;
determining, based on a machine learning model, a three-dimensional feature associated with the sensor data,
wherein the machine learning model is trained using a training dataset comprising a determined ground truth and corresponding sensor data captured within a period of time,
wherein the machine learning model is trained to output the determined ground truth based on an input of at least a portion of the corresponding sensor data comprising a particular time series element of the plurality of time series elements, and
wherein the determined ground truth is indicative of a three-dimensional feature associated with the corresponding sensor data; and

adjusting operation of the vehicle based on the three-dimensional feature.

11. The method of claim 10 , wherein the three-dimensional feature is a three-dimensional trajectory of a real-world feature.

12. The method of claim 11 , wherein the real-world feature is a vehicle lane line.

13. The method of claim 11 , wherein the real-world feature is a different vehicle.

14. The method of claim 10 , wherein adjusting operation of the vehicle comprises causing adjustment of a speed or steering of the vehicle.

15. The method of claim 10 , wherein the ground truth is determined via selecting portions of each of the time series elements which are associated with a highest certainty in depicting respective portions of a real-world feature.

16. The method of claim 15 , wherein the real-world feature is a vehicle lane line and wherein at least one unselected portion occludes the vehicle lane line.

17. The method of claim 16 , wherein the ground truth is determined based on the selected portions and elevation information associated with the vehicle lane line.

18. A computer program product, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions which when executed by a system of one or more processors, cause the one or more processors to:
obtain sensor data via one or more sensors of a vehicle;
determine, based on a machine learning model, a three-dimensional feature associated with the sensor data,
wherein the machine learning model is trained using a training dataset comprising a determined ground truth and corresponding sensor data captured within a period of time, the corresponding sensor data comprising a plurality of time series elements
wherein the machine learning model is trained to output the determined ground truth based on an input of at least a portion of the corresponding sensor data comprising a particular time series element of the plurality of time series elements, and
wherein the determined ground truth is indicative of a three-dimensional feature associated with the corresponding sensor data; and

adjust operation of the vehicle based on the three-dimensional feature.

Assignees

• Tesla Inc

Inventors

• Elluswamy Ashok Kumar
• Bauch Matthew
• Payne Christopher
• Karpathy Andrej
• Shroff Dhaval
• Ramanandan Arvind
• Hakewill James Robert Howard

Classifications

• G06N3/08

  Learning methods · CPC title

• G05D1/617

  Safety or protection, e.g. defining protection zones around obstacles or avoiding hazards (arrangements for controlling the position or course of two or more vehicles for avoiding collisions therebetween G05D1/693; arrangements for reacting to or preventing system or operator failure G05D1/80) · CPC title

• G05D1/249

  from positioning sensors located off-board the vehicle, e.g. from cameras · CPC title

• G05D1/65

  Following a desired speed profile · CPC title

• G06V10/82

  using neural networks · CPC title