Apparatus and methodology of road condition classification using sensor data

Patent metadata

Field	Value
Publication number	US-11492006-B2
Application number	US-202016944911-A
Country	US
Kind code	B2
Filing date	Jul 31, 2020
Priority date	Jul 31, 2020
Publication date	Nov 8, 2022
Grant date	Nov 8, 2022

Abstract

Methods and systems are provided for controlling a vehicle action based on a condition of a road on which a vehicle is travelling, including: obtaining first sensor data as to a surface of the road from one or more first sensors onboard the vehicle; obtaining second sensor data from one or more second sensors onboard the vehicle as to a measured parameter pertaining to operation of the vehicle or conditions pertaining thereto; generating a plurality of road surface channel images from the first sensor data, wherein each road surface channel image captures one of a plurality of facets of properties of the first sensor data; classifying, via a processor using a neural network model, the condition of the road on which the vehicle is travelling, based on the measured parameter and the plurality of road surface channel images; and controlling a vehicle action based on the classification of the condition of the road.

First claim

What is claimed is:

1. A method for controlling a vehicle action based on a condition of a road on which a vehicle is travelling, the method comprising:
obtaining first sensor data as to a surface of the road from one or more first sensors onboard the vehicle;
obtaining second sensor data from one or more second sensors onboard the vehicle as to a measured parameter pertaining to operation of the vehicle or conditions pertaining thereto;
generating a plurality of road surface channel images from the first sensor data, wherein each road surface channel image captures one of a plurality of facets of properties of the first sensor data;
classifying, via a processor using a neural network model, the condition of the road on which the vehicle is travelling, based on the measured parameter and the plurality of road surface channel images; and
controlling a vehicle action based on the classification of the condition of the road.

2. The method of claim 1 , wherein the parameter comprises a speed of the vehicle.

3. The method of claim 1 , wherein the plurality of facets of properties of the first sensor data, as reflected in the surface channel images, include returned energy at an (x,y) position from the first sensors.

4. The method of claim 1 , wherein the plurality of facets of properties of the first sensor data, as reflected in the surface channel images, include a Z coordinate at an (x,y) position from the first sensors.

5. The method of claim 1 , wherein the plurality of facets of properties of the first sensor data, as reflected in the surface channel images, include a Doppler value at an (x,y) position from the first sensors.

6. The method of claim 1 , wherein the plurality of facets of properties of the first sensor data, as reflected in the surface channel images, include a sensor index value at an (x,y) position from the first sensors.

7. The method of claim 1 , wherein the plurality of facets of properties of the first sensor data, as reflected in the surface channel images, include each of the following: (i) returned energy at an (x,y) position from the first sensors; (ii) a Z coordinate at the (x,y) position from the first sensors; (iii) a Doppler value at the (x,y) position from the first sensors; and (iv) a sensor index value at the (x,y) position from the first sensors.

8. The method of claim 2 , further comprising:
generating, via the processor, a speed image channel based on a categorization of the vehicle speed; and
fusing, via the processor, the plurality of road surface channel images with the speed image channel;
wherein the step of classifying the condition comprises classifying, via the processor using a neural network model, the condition of the road on which the vehicle is travelling, based on the fusing of the plurality of road surface channel images with the speed image channel.

9. The method of claim 2 , further comprising:
performing, via the processor, feature extraction from the plurality of road surface channel images; and
performing feature level fusion between a categorization of the vehicle speed and the feature extraction form the plurality of road surface channel images;
wherein the step of classifying the condition comprises classifying, via the processor using a neural network model, the condition of the road on which the vehicle is travelling, based on the feature level fusion between a categorization of the vehicle speed and the feature extraction form the plurality of road surface channel images.

10. A system for controlling a vehicle action based on classifying a condition of a road on which a vehicle is travelling, the system comprising:
one or more first sensors configured to provide first sensor data as to a surface of the road;
one or more second sensors configured to provide second sensor data as to a measured parameter pertaining to operation of the vehicle or conditions pertaining thereto; and
a processor disposed coupled to the first sensors and the second sensors and configured to:
generate a plurality of road surface channel images from the first sensor data, wherein each road surface channel image captures one of a plurality of facets of properties of the first sensor data;
classify, using a neural network model, the condition of the road on which the vehicle is travelling, based on the measured parameter and the plurality of road surface channel images; and
provide instructions to control a vehicle action based on the condition of the road.

11. A vehicle comprising:
a body;
a drive system disposed within the body and configured to drive the vehicle; and
a control system coupled to the drive system, the control system comprising:
one or more first sensors configured to provide first sensor data as to a surface of the road;
one or more second sensors configured to provide second sensor data as to a measured parameter pertaining to operation of the vehicle or conditions pertaining thereto; and
a processor disposed coupled to the first sensors and the second sensors and configured to:
generate a plurality of road surface channel images from the first sensor data, wherein each road surface channel image captures one of a plurality of facets of properties of the first sensor data;
classify, using a neural network model, the condition of the road on which the vehicle is travelling, based on the measured parameter and the plurality of road surface channel images; and
provide instructions to control a vehicle action based on the condition of the road.

12. The vehicle of claim 11 , wherein the parameter comprises a speed of the vehicle.

13. The vehicle of claim 12 , wherein:
the vehicle includes a front bumper; and
the one or more first sensors comprise a plurality of ultra-short range radar (USRR) sensors disposed proximate the front bumper of the vehicle.

14. The vehicle of claim 11 , wherein the condition comprises a surface condition of a surface of the road, as to whether the surface is wet, dry, or covered with snow.

15. The vehicle of claim 11 , wherein the condition comprises a material of which a surface of the road is made.

16. The vehicle of claim 11 , wherein the plurality of facets of properties of the first sensor data, as reflected in the surface channel images, include returned energy at an (x,y) position from the first sensors.

17. The vehicle of claim 11 , wherein the plurality of facets of properties of the first sensor data, as reflected in the surface channel images, include a Doppler value at an (x,y) position from the first sensors.

18. The vehicle of claim 11 , wherein the plurality of facets of properties of the first sensor data, as reflected in the surface channel images, include a sensor index value at an (x,y) position from the first sensors.

19. The vehicle of claim 12 , wherein the processor is further configured to:
generate a speed image channel based on a categorization of the vehicle speed;
fuse the plurality of road surface channel images with the speed image channel; and
classify, using a neural network model, the condition of the road on which the vehicle is travelling, based on the fusing of the plurality of road surface channel images with the speed image channel.

20. The vehicle of claim 12 , wherein the processor is further configured to:
perform feature extraction from the plurality of road surface channel images;
perform feature level fusion between a categorization of the vehicle speed and the feature extraction form the plurality of road surface channel images; and
classify, using a neu

Assignees

• Gm Global Tech Operations Llc

Inventors

• Zhao Qingrong
• Litkouhi Bakhtiar B
• Khalili Mohsen

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06F18/24

  Classification techniques · CPC title

• B60W2520/10

  Longitudinal speed · CPC title

• G01S2013/93271

  in the front of the vehicles · CPC title

• G06V10/764 Primary

  using classification, e.g. of video objects · CPC title