Road condition deep learning model

Patent metadata

Field	Value
Publication number	US-11775870-B2
Application number	US-202217978287-A
Country	US
Kind code	B2
Filing date	Nov 1, 2022
Priority date	Jun 5, 2020
Publication date	Oct 3, 2023
Grant date	Oct 3, 2023

Abstract

The technology relates to using on-board sensor data, off-board information and a deep learning model to classify road wetness and/or to perform a regression analysis on road wetness based on a set of input information. Such information includes on-board and/or off-board signals obtained from one or more sources including on-board perception sensors, other on-board modules, external weather measurement, external weather services, etc. The ground truth includes measurements of water film thickness and/or ice coverage on road surfaces. The ground truth, on-board and off-board signals are used to build the model. The constructed model can be deployed in autonomous vehicles for classifying/regressing the road wetness with on-board and/or off-board signals as the input, without referring to the ground truth. The model can be applied in a variety of ways to enhance autonomous vehicle operation, for instance by altering current driving actions, modifying planned routes or trajectories, activating on-board cleaning systems, etc.

First claim

The invention claimed is:

1. A method for generating a road condition deep learning model, the method comprising:
obtaining as a first set of training inputs, by one or more processors, sensor data of an environment along a portion of a roadway;
obtaining as a second set of training inputs, by the one or more processors, information associated with the portion of the roadway:
evaluating, by the one or more processors, the first set of training inputs and the second set of training inputs with respect to wetness ground truth data for the portion of the roadway, to classify or estimate wetness along one or more areas of the portion of the roadway:
generating the road condition deep learning model based upon the evaluation; and
storing the generated road condition deep learning model in memory.

2. The method of claim 1 , wherein the obtained sensor data includes one or more of lidar returns, camera imagery, radar returns, audio signals, or output from a vehicle on-board module.

3. The method of claim 1 , wherein the information associated with the portion of the roadway includes one or more of weather station information, a weather forecast, road graph data, crowdsourced information, or observations from one or more road users.

4. The method of claim 1 , further comprising performing signal fusion of some or all of the obtained sensor data and off-board information.

5. The method of claim 1 , further comprising applying weighting to different signals of the obtained sensor data.

6. The method of claim 1 , further comprising, prior to generating the road condition deep learning model:
performing a statistical analysis to determine which obtained sensor data correlates with the ground truth data:
removing information of one or more objects on the roadway from the obtained sensor data; or
limiting the obtained sensor data to a selected range or distance.

7. The method of claim 6 , further comprising deemphasizing any received sensor data that does not meet a correlation threshold with the wetness ground truth data.

8. The method of claim 1 , further comprising smoothing ground truth measurements of the wetness ground truth data prior to generating the road condition deep learning model.

9. The method of claim 1 , further comprising applying the road condition deep learning model to one or more roadway regions to either identify a probability of wetness for each of the regions or estimate water film depth for each of the region.

10. The method of claim 1 , further comprising causing one or more systems of a self-driving vehicle to perform at least one of altering a current driving action, modifying a planned route or trajectory, or activating an on-board cleaning system of the self-driving vehicle based on the road condition deep learning model output.

11. The method of claim 1 , wherein storing the generated road condition deep learning model in memory comprises storing the generated road condition deep learning model in memory of one or more vehicles that are not equipped with a ground truth measurement sensor, the stored road condition deep learning model configured for use in evaluating real-time road wetness.

12. A method comprising:
receiving sensor data including imagery of a roadway, the sensor data having been captured by a vehicle driving on the roadway;
obtaining map information of the roadway, the map information comprising at least one of geometry information or lane information of the roadway;
performing regression analysis using a trained neural network using the imagery and the map information of the roadway; and
determining an estimated snow depth for one or more lanes of the roadway based on an output of the regression analysis;
wherein the neural network is trained with training data including labeled road wetness ground truth examples.

13. The method of claim 12 , further comprising operating the vehicle in an autonomous driving mode based on determining the estimated snow depth for the one or more lanes of the roadway.

14. A method comprising:
obtaining, by one or more processors, sensor data from one or more sensors of a perception system of a vehicle, the one or more sensors being configured to detect objects or conditions in an environment around the vehicle:
using, by the one or more processors, a stored a road condition deep learning model to generate information associated with a discrete classification or continuous regression/estimation of road wetness based on the obtained sensor data; and
using, by the one or more processors, the generated information to issue a control signal associated with operation of the vehicle.

15. The method of claim 14 , wherein the control signal is configured to trigger a safety precaution for operation of the vehicle in an autonomous driving mode.

16. The method of claim 14 , wherein the control signal is configured to alter a current driving action of the vehicle in an autonomous driving mode.

17. The method of claim 14 , wherein the control signal is configured to modify a trajectory for the vehicle.

18. The method of claim 14 , wherein the control signal is configured to activate an on-board cleaning system.

19. The method of claim 14 , wherein the control signal is configured to modify a threshold for one or more aspects of the perception system.

20. The method of claim 19 , wherein modification of the threshold for one or more aspects of the perception system includes modification of at least one of a threshold for filtering, a sensor noise level, a sensor field of view adaptation, a sensor validation logic, or a pedestrian detector.

21. The method of claim 20 , wherein the control signal is configured to change or update a model for predicting behavior of other road users.

22. The method of claim 20 , wherein the control signal is configured to modify a planned route for the vehicle.

23. The method of claim 22 , wherein modifying a planned route includes changing at least one of a pickup location, a drop-off location, or a lane of travel.

Assignees

• Waymo Llc

Inventors

• Zhou Xin
• Cooper Roshni
• James Michael

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/0985

  Hyperparameter optimisation; Meta-learning; Learning-to-learn · CPC title

• G06N3/09

  Supervised learning · CPC title

• G05D1/247

  using signals provided by artificial sources external to the vehicle, e.g. navigation beacons · CPC title

• G05D1/249

  from positioning sensors located off-board the vehicle, e.g. from cameras · CPC title