Virtual sensor-data-generation system and method supporting development of vision-based rain-detection algorithms

Patent metadata

Field	Value
Publication number	US-10521677-B2
Application number	US-201615210670-A
Country	US
Kind code	B2
Filing date	Jul 14, 2016
Priority date	Jul 14, 2016
Publication date	Dec 31, 2019
Grant date	Dec 31, 2019

Abstract

A method for generating training data is disclosed. The method may include executing a simulation process. The simulation process may include traversing a virtual camera through a virtual driving environment comprising at least one virtual precipitation condition and at least one virtual no precipitation condition. During the traversing, the virtual camera may be moved with respect to the virtual driving environment as dictated by a vehicle-motion model modeling motion of a vehicle driving through the virtual driving environment while carrying the virtual camera. Virtual sensor data characterizing the virtual driving environment in both virtual precipitation and virtual no precipitation conditions may be recorded. The virtual sensor data may correspond to what a real sensor would have output had it sensed the virtual driving environment in the real world.

First claim

What is claimed is:

1. A method for generating a plurality of electronically annotated digital image files suitable for use as training data in a machine learning process, the method comprising:
traversing, by a computer system, one or more virtual cameras over a virtual road surface in a simulation;
recording, by the computer system, a plurality of electronic digital image files corresponding to signals output by the one or more virtual cameras during the traversing; and
converting, by the computer system, the plurality of electronic digital image files to training data by electronically annotating each electronic digital image file thereof with ground-truth data indicating whether virtual precipitation from the simulation is encoded therewithin.

2. The method of claim 1 , wherein the annotating comprises annotating each electronic digital image file of the plurality of electronic digital image files with ground-truth data indicating whether virtual rain from the simulation is encoded therewithin.

3. The method of claim 2 , wherein the traversing comprises moving each of the one or more virtual cameras with respect to the virtual road surface as dictated by a vehicle-motion model modeling motion of a vehicle carrying the one or more virtual cameras and driving on the virtual road surface.

4. The method of claim 3 , wherein the traversing comprises traversing in the simulation the one or more virtual cameras over the virtual road surface and through at least one virtual rain condition and at least one virtual no rain condition.

5. The method of claim 4 , further comprising using the training data to train an artificial neural network to distinguish between photographic data corresponding to the rain condition and photographic data corresponding to the no rain condition.

6. The method of claim 5 , wherein the one or more virtual cameras comprise a forward-looking camera positioned to sense a portion of the virtual road surface ahead of the vehicle.

7. The method of claim 5 , wherein the one or more virtual cameras comprise a rearward-looking camera positioned to sense a portion of the virtual road surface behind the vehicle.

8. The method of claim 1 , wherein the traversing comprises moving each of the one or more virtual cameras with respect to the virtual road surface as dictated by a vehicle-motion model modeling motion of a vehicle carrying the one or more virtual cameras and driving on the virtual road surface.

9. The method of claim 8 , wherein the one or more virtual cameras comprise a forward-looking camera positioned to sense a portion of the virtual road surface ahead of the vehicle.

10. The method of claim 8 , wherein the one or more virtual cameras comprise a rearward-looking camera positioned to sense a portion of the virtual road surface behind the vehicle.

11. The method of claim 1 , further comprising using the training data to train an artificial neural network to distinguish between photographic data corresponding to a precipitation condition and photographic data corresponding to the no precipitation condition.

12. A method for generating a plurality of electronically annotated digital image files suitable for use as training data in a machine learning process, the method comprising:
executing, by a computer system, a simulation comprising
traversing one or more virtual cameras over a virtual road surface and through at least one virtual precipitation condition and at least one virtual no precipitation condition that are sensible by the one or more virtual cameras, and
moving, during the traversing, each of the one or more virtual cameras with respect to the virtual road surface as dictated by a vehicle-motion model modeling motion of a vehicle driving on the virtual road surface while carrying the one or more virtual cameras;

recording, by the computer system, a plurality of electronic digital image files characterizing the at least one virtual precipitation condition and at least one virtual no precipitation condition, the plurality of electronic digital image files corresponding to signal output by the one or more virtual cameras during the traversing; and
converting, by the computer system, the plurality of electronic digital image files to training data by electronically annotating each electronic digital image file thereof with ground-truth data indicating whether the virtual precipitation or the virtual no precipitation condition is encoded therewithin.

13. The method of claim 12 , further comprising using the training data to train an artificial neural network to distinguish between photographic data corresponding to a rain condition and photographic data corresponding to the no rain condition.

14. The method of claim 12 , wherein the one or more virtual cameras comprise a forward-looking camera positioned to sense a portion of the virtual road surface ahead of the vehicle.

15. The method of claim 12 , wherein the one or more virtual cameras comprise a rearward-looking camera positioned to sense a portion of the virtual road surface behind the vehicle.

16. A computer system comprising:
one or more processors;
memory operably connected to the one or more processors; and
the memory storing
a virtual driving environment programmed to include at least one virtual precipitation condition and at least one virtual no precipitation condition,
a first software model programmed to model a digital camera,
a second software model programmed to model a vehicle,
a simulation module programmed to use the virtual driving environment, the first software model, and the second software model to produce an a plurality of electronic digital image files modeling what would be output by the digital camera had the digital camera been mounted to the vehicle and the vehicle had driven on an actual driving environment matching the virtual driving environment, and
the simulation module further programmed to electronically annotate each electronic digital image file of the plurality of electronic digital image files with ground-truth data indicating whether the at least one virtual precipitation condition or the at least one virtual no precipitation condition is encoded therewithin.

17. The system of claim 16 , wherein the digital camera comprises a forward-looking camera positioned to sense a portion of the virtual driving environment ahead of the vehicle.

18. The system of claim 16 , wherein the digital camera comprises a rearward-looking camera positioned to sense a portion of the virtual driving environment behind the vehicle.

Assignees

• Ford Global Tech Llc

Inventors

• Micks Ashley Elizabeth
• Banvait Harpreetsingh
• Jain Jinesh J
• Nariyambut Murali Vidya

Classifications

• G06V20/58

  Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads · CPC title

• G06F30/15 Primary

  Vehicle, aircraft or watercraft design · CPC title

• G06F18/24

  Classification techniques · CPC title

• H04N23/90

  Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums · CPC title

• G06F18/28

  Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries · CPC title