Audio-based technique to sense and detect the road condition for autonomous driving vehicles

Patent metadata

Field	Value
Publication number	US-12043263-B2
Application number	US-202117148254-A
Country	US
Kind code	B2
Filing date	Jan 13, 2021
Priority date	Jan 13, 2021
Publication date	Jul 23, 2024
Grant date	Jul 23, 2024

Abstract

Sound signals are received by one or more microphones disposed at an ADV. The sound signals are analyzed to extract a feature of a road on which the ADV is driving. A road condition of the road is determined based on the extracted feature. A path planning and speed planning is performed to generate a trajectory based on the road condition. The ADV is controlled to drive autonomously according to the generated trajectory.

First claim

What is claimed is:

1. A computer-implemented method for operating an autonomous driving vehicle (ADV), the method comprising:
receiving sound signals by one or more microphones disposed at the ADV, comprising:
detecting the sound signals from a friction between each wheel of the ADV and a surface of a road, at least two microphones of the one or more microphones being mounted at a hub of each wheel of the ADV, the sound signals from the at least two microphones having a reduced noise by applying a noise reducing algorithm;

analyzing the sound signals to extract a feature representing the road on which the ADV is driving, comprising:
extracting a power spectrogram of the road based on the sound signals, the power spectrogram representing a power level of a signal of the sound signals in each frequency band over time and corresponding to a road condition of the road;

determining the road condition of the road based on the extracted feature, comprising:
determining a pavement condition of the road based on the extracted feature, the pavement condition including an asphalt condition, a rocky condition, or a sandy condition;
determining a depth of a covering of the road based on the extracted feature, the depth of the covering of the road including a depth of water, snow, or sand on the road; and
determining a friction coefficient of the road based on at least one of a surface type, the pavement condition, or the depth of the covering of the road by using regression models in a machine learning algorithm;

performing a path planning and speed planning to generate a trajectory based on the road condition, comprising:
performing at least one of changing a path or lowering a speed of the ADV, in response to that the surface type is a wet surface and the friction coefficient is less than a predetermined threshold; and

controlling the ADV to drive autonomously according to the generated trajectory.

2. The method of claim 1 , wherein the one or more microphones are disposed inside tires or wheel hubs.

3. The method of claim 1 , wherein the one or more microphones are disposed at a bottom panel of the ADV.

4. The method of claim 1 , wherein analyzing the sound signals to extract the feature of the road comprises analyzing the sound signals to extract a frequency feature representing a roughness of the surface of the road.

5. The method of claim 1 , wherein determining the road condition comprises determining the road condition using the machine learning algorithm or a deep learning algorithm.

6. The method of claim 1 , wherein determining the road condition based on the extracted feature comprises determining at least one of a surface type of the road or the pavement condition based on the extracted feature.

7. The method of claim 1 , wherein determining the road condition based on the extracted feature comprises determining the friction coefficient representing a friction between one or more tires of the ADV and the surface of the road based on the extracted feature.

8. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations of operating an autonomous driving vehicle (ADV), the operations comprising:
receiving sound signals by one or more microphones disposed at the ADV, comprising:
detecting the sound signals from a friction between each wheel of the ADV and a surface of a road, at least two microphones of the one or more microphones being mounted at a hub of each wheel of the ADV, the sound signals from the at least two microphones having a reduced noise by applying a noise reducing algorithm;

analyzing the sound signals to extract a feature representing the road on which the ADV is driving, comprising:
extracting a power spectrogram of the road based on the sound signals, the power spectrogram representing a power level of a signal of the sound signals in each frequency band over time and corresponding to a road condition of the road;

determining the road condition of the road based on the extracted feature, comprising:
determining a pavement condition of the road based on the extracted feature, the pavement condition including an asphalt condition, a rocky condition, or a sandy condition;
determining a depth of a covering of the road based on the extracted feature, the depth of the covering of the road including a depth of water, snow, or sand on the road; and
determining a friction coefficient of the road based on at least one of a surface type, the pavement condition, or the depth of the covering of the road by using regression models in a machine learning algorithm;

performing a path planning and speed planning to generate a trajectory based on the road condition, comprising:
performing at least one of changing a path or lowering a speed of the ADV, in response to that the surface type is a wet surface and the friction coefficient is less than a predetermined threshold; and

controlling the ADV to drive autonomously according to the generated trajectory.

9. The machine-readable medium of claim 8 , wherein the one or more microphones are disposed inside tires or wheel hubs.

10. The machine-readable medium of claim 8 , wherein the one or more microphones are disposed at a bottom panel of the ADV.

11. The machine-readable medium of claim 8 , wherein analyzing the sound signals to extract the feature of the road comprises analyzing the sound signals to extract a frequency feature representing a roughness of the surface of the road.

12. The machine-readable medium of claim 8 , wherein determining the road condition comprises determining the road condition using the machine learning algorithm or a deep learning algorithm.

13. The machine-readable medium of claim 8 , wherein determining the road condition based on the extracted feature comprises determining at least one of a surface type of the road or the pavement condition based on the extracted feature.

14. The machine-readable medium of claim 8 , wherein determining the road condition based on the extracted feature comprises determining the friction coefficient representing a friction between one or more tires of the ADV and the surface of the road based on the extracted feature.

15. A data processing system, comprising:
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations of operating an autonomous driving vehicle (ADV), the operations including
receiving sound signals by one or more microphones disposed at the ADV, comprising:
detecting the sound signals from a friction between each wheel of the ADV and a surface of a road, at least two microphones of the one or more microphones being mounted at a hub of each wheel of the ADV, the sound signals from the at least two microphones having a reduced noise by applying a noise reducing algorithm;

analyzing the sound signals to extract a feature representing the road on which the ADV is driving, comprising:
extracting a power spectrogram of the road based on the sound signals, the power spectrogram representing a power level of a signal of the sound signals in each frequency band over time and corresponding to a road condition of the road;

determining the road condition of the road based on the extracted feature, comprising:
determining a pavement condition of the road based on the extracted feature, the pavement condition including an asphalt condition, a rocky condition, or a sandy condition;
determining a depth of a covering of the road based on the extracted feature, the depth of the coverin

Assignees

• Baidu Usa Llc

Inventors

• Wang Wei
• Luo Qi
• Xu Kecheng
• Sun Hongyi
• Reynolds Wesley
• Zou Youling
• Lin Zejun

Classifications

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/09

  Supervised learning · CPC title

• H04S7/40

  Visual indication of stereophonic sound image · CPC title

• H04R1/222

  for microphones (H04R1/24, H04R1/26 take precedence) · CPC title

• G06N3/08

  Learning methods · CPC title