Generating simulated sensor data for training and validation of detection models

Patent metadata

Field	Value
Publication number	US-10228693-B2
Application number	US-201715406031-A
Country	US
Kind code	B2
Filing date	Jan 13, 2017
Priority date	Jan 13, 2017
Publication date	Mar 12, 2019
Grant date	Mar 12, 2019

Abstract

A scenario is defined that including models of vehicles and a typical driving environment. A model of a subject vehicle is added to the scenario and sensor locations are defined on the subject vehicle. Perception of the scenario by sensors at the sensor locations is simulated to obtain simulated sensor outputs. The simulated sensor outputs are annotated to indicate the location of obstacles in the scenario. The annotated sensor outputs may then be used to validate a statistical model or to train a machine learning model. The simulates sensor outputs may be modeled with sufficient detail to include sensor noise or may include artificially added noise to simulate real world conditions.

First claim

The invention claimed is:

1. A method comprising, by a computer system:
simulating perception of a 3D model by one or more sensors to obtain one or more sensor outputs such that the one or more sensor outputs simulate sensor noise;
annotating the one or more sensor outputs according to locations of obstacles in the 3D model; and
at least one of training and testing a model according to the one or more sensor outputs and the annotations;
wherein simulating perception of the 3D model by the one or more sensors to obtain the one or more sensor outputs such that the one or more sensor outputs simulate the sensor noise comprises:
identifying a location of an obstacle in the 3D model;
generating one or more sensor outputs corresponding to the location of the obstacle relative to locations of the one or more sensors in the 3D model; and
adding noise to the one or more sensor outputs according to one or more models of variances of the one or more sensors.

2. The method of claim 1 , wherein the one or more sensors are defined with respect to model of a subject vehicle;
wherein the one or more sensors are defined by one or more camera locations; and
wherein simulating perception of the 3D model by the one or more sensors comprises simulating detection of images of the 3D model from the one or more camera locations.

3. The method of claim 1 , wherein the one or more sensors are defined with respect to a model of a subject vehicle;
wherein the one or more sensors are defined by a RADAR (radio detection and ranging) sensor location; and
wherein simulating perception of the 3D model by the one or more sensors comprises simulating a RADAR sensor output according to perception of the 3D model from the RADAR sensor location.

4. The method of claim 1 , wherein the one or more sensors are defined with respect to a model of a subject vehicle;
wherein the one or more sensors are defined by a LIDAR (light detection and ranging) sensor location; and
wherein simulating perception of the 3D model by the one or more sensors comprises simulating a LIDAR sensor output according to perception of the 3D model from the LIDAR sensor location.

5. The method of claim 1 , wherein the 3D model further includes a definition of velocities for one or more adjacent vehicles and a velocity of a subject vehicle defining one or more locations of the one or more sensors.

6. The method of claim 1 , wherein annotating the one or more sensor outputs according to the locations of obstacles in the 3D model comprises annotating the one or more sensor outputs with the locations of the obstacles in the 3D model; and
wherein at least one of training and testing the model according to the one or more sensor outputs and the annotations comprises testing a statistical model that tracks the obstacles and assigns a probability to an expected location for each obstacle of the obstacles.

7. The method of claim 1 , wherein annotating the one or more sensor outputs according to the locations of obstacles in the 3D model comprises annotating the one or more sensor outputs with an occupancy status of a region adjacent a model of a subject vehicle according to the locations of the obstacles in the 3D model;
wherein at least one of training and testing the model according to the one or more sensor outputs and the annotations comprises testing a statistical model that updates a probability of occupancy of the region according to the one or more sensor outputs.

8. The method of claim 1 , wherein annotating the one or more sensor outputs according to the locations of obstacles in the 3D model comprises annotating the one or more sensor outputs with a grid such that each square of the grid is annotated with whether the each square is occupied by one of the obstacles;
wherein at least one of training and testing the model according to the one or more sensor outputs and the annotations the statistical model comprises testing a statistical model that updates a probability of occupancy of squares of the grid surrounding a vehicle according to the one or more sensor outputs.

9. A system comprising one or more processing devices and one or more memory devices operably coupled to the one or more memory devices, the one or more memory devices storing executable code effective to cause the one or more processing devices to:
define a three-dimensional (3D) model including a subject vehicle defining one or more sensor locations and one or more obstacles;
simulate perception of the 3D model by one or more sensors at the one or more sensor locations to obtain one or more sensor outputs such that the one or more sensor outputs simulate sensor noise;
annotate the one or more sensor outputs according to locations of obstacles in the 3D model; and
at least one of train and test a model according to the one or more sensor outputs and the annotations;
wherein the executable code is further effective to cause the one or more processors to simulate perception of the 3D model by the one or more sensors by simulating a RADAR (radio detection and ranging) sensor output according to perception of the 3D model from the one or more sensor locations; and
wherein the executable code is further effective to cause the one or more processors to simulate perception of the 3D model by the one or more sensors in sufficient detail that the sensor noise results from multiple simulated reflections of a simulated electromagnetic wave propagated from a simulated RADAR sensor.

10. The system of claim 9 , wherein the 3D model further includes definition of velocities for one or more adjacent vehicles and a velocity of a subject vehicle defining one or more locations of the one or more sensors.

11. The system of claim 9 , wherein the executable code is further effective to cause the one or more processors to:
annotate the one or more sensor outputs according to the locations of obstacles in the 3D model by annotating the one or more sensor outputs with the locations of the obstacles in the 3D model; and
at least one of train and test the model according to the one or more sensor outputs and the annotations by testing a statistical model that tracks the obstacles and assigns a probability to an expected location for each obstacle of the obstacles.

12. The system of claim 9 , wherein the executable code is further effective to cause the one or more processors to:
annotate the one or more sensor outputs according to the locations of obstacles in the 3D model by annotating the one or more sensor outputs with an occupancy status of a region adjacent a model of a subject vehicle according to the locations of the obstacles in the 3D model;
at least one of train and test the model according to the one or more sensor outputs and the annotations by testing a statistical model that updates a probability of occupancy of the region according to the one or more sensor outputs.

13. The system of claim 9 , wherein the executable code is further effective to cause the one or more processors to:
annotate the one or more sensor outputs according to the locations of obstacles in the 3D model by annotating the one or more sensor outputs with a grid such that each square of the grid is annotated with whether the each square is occupied by one of the obstacles;
at least one of train and test the model according to the one or more sensor outputs and the annotations by testing a statistical model that updates a probability of occupancy of squares of the grid surrounding a vehicle according to the one or more sensor outputs.

Assignees

• Ford Global Tech Llc

Inventors

• Micks Ashley Elizabeth
• Kadetotad Sneha
• Jain Jinesh J
• Liu Dongran
• Gerardo Castro Marcos Paul
• Murali Vidya Nariyambut

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06F18/29

  Graphical models, e.g. Bayesian networks · CPC title

• G06F30/15

  Vehicle, aircraft or watercraft design · CPC title

• G06F30/20 Primary

  Design optimisation, verification or simulation (optimisation, verification or simulation of circuit designs G06F30/30) · CPC title

• G08G1/16

  Anti-collision systems (road vehicle drive control systems for predicting or avoiding probable or impending collision otherwise than by control of a particular sub-unit B60W30/08) · CPC title