Neural network audio processing to determine weather characteristics

Patent metadata

Field	Value
Publication number	US-2025206316-A1
Application number	US-202318392371-A
Country	US
Kind code	A1
Filing date	Dec 21, 2023
Priority date	Dec 21, 2023
Publication date	Jun 26, 2025
Grant date	—

Abstract

A method includes receiving an audio signal from an audio sensor of a vehicle, determining an audio feature based at least in part on the received audio signal, using a neural network to process the determined audio feature to estimate a characteristic of weather in an environment of the vehicle, and modifying a parameter of the vehicle based at least in part on the estimated characteristic.

First claim

What is claimed is:

1 . A vehicle comprising:
a microphone; one or more processors; and one or more non-transitory computer-readable media storing instructions which, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving an audio signal from the microphone;
applying a short-time Fourier transform (STFT) to the received audio signal to generate a spectrogram of the received audio signal;
processing, using a convolutional neural network (CNN), at least part of the generated spectrogram to determine a number of raindrops striking the vehicle in a vicinity of the microphone within an interval;
estimating, based at least in part on the determined number of raindrops, a characteristic associated with an environment of the vehicle;
determining, based at least in part on the estimated characteristic, that the environment is outside of an approved operational design domain (ODD) for the vehicle; and
modifying, based at least in part on the determining that the environment is outside of the approved ODD, a parameter of the vehicle.

2 . The vehicle of claim 1 , wherein:
the operations comprise obtaining data indicative of a speed of the vehicle within the interval; and the estimating of the characteristic comprises estimating, using the data indicative of the speed of the vehicle and the determined number of raindrops striking the vehicle in within the interval, the absolute rain rate.

3 . The vehicle of claim 1 , wherein:
the microphone is a first microphone disposed at a first position on the vehicle; the audio signal is a first audio signal; the operations comprise receiving a second audio signal from a second microphone disposed at a second position on the vehicle; and the estimating of the characteristic is further based on the second audio signal.

4 . The vehicle of claim 1 , wherein the operations comprise suspending full autonomous driving operations based at least in part on the modifying of the parameter.

5 . A method comprising:
receiving an audio signal from an audio sensor of a vehicle; determining an audio feature from the received audio signal; processing, using a neural network, the audio feature to estimate a characteristic of weather in an environment of the vehicle; and modifying, based at least in part on the estimated characteristic, a parameter of the vehicle.

6 . The method of claim 5 , comprising obtaining data indicative of a speed of the vehicle,
wherein the processing comprises at least one of:
processing an output of the neural network together with the data indicative of the speed of the vehicle, or
processing, using the neural network, the data indicative of the speed of the vehicle together with the audio feature.

7 . The method of claim 5 , comprising obtaining data indicative of an environmental condition, wherein:
the processing comprises processing, using the neural network, the data indicative of the environmental condition together with the audio feature; and the environmental condition comprises any of wind velocity, humidity, air temperature, or air pressure.

8 . The method of claim 5 , wherein:
the transducer is a first transducer disposed at a first position on the vehicle; the audio signal is a first audio signal; the method comprises receiving a second audio signal from a second transducer disposed at a second position on the vehicle; and the estimating of the characteristic is further based on the second audio signal.

9 . The method of claim 5 , comprising controlling the vehicle according to the modified parameter of the vehicle.

10 . The method of claim 5 , comprising
determining, based at least in part on the estimated characteristic, that the environment is outside of an approved ODD for the vehicle; and suspending full autonomous driving operations based at least in part on the modifying of the parameter.

11 . The method of claim 5 , wherein the neural network is a fine-tuned neural network, the method comprising:
obtaining a set of base layers of a pre-trained neural network, wherein the pre-trained neural network has been trained to detect audio events; adding a network head to the set of base layers to obtain an intermediate neural network; and training the intermediate neural network using data comprising training audio features with respective labels indicating respective characteristics associated with the training audio features, thereby to obtain the fine-tuned neural network.

12 . The method of claim 5 , wherein the characteristic is indicative of a rate of precipitation events.

13 . The method of claim 5 , wherein the characteristic comprises a classification of a type of precipitation in the environment of the vehicle.

14 . The method of claim 5 , wherein the audio sensor is a microphone.

15 . One or more non-transitory computer-readable media storing instructions executable by one or more processors, wherein the instructions, when executed, cause the one or more processors to perform operations comprising:
receiving an audio signal from an audio sensor of a vehicle; determining an audio feature from the received audio signal; processing, using a neural network, the audio feature to estimate a characteristic of weather in an environment of the vehicle; and modifying, based at least in part on the estimated characteristic, a parameter of the vehicle.

16 . The one or more non-transitory computer-readable media of claim 15 , wherein:
the operations comprise obtaining data indicative of a speed of the vehicle; and the processing comprises at least one of:
processing an output of the neural network together with the data indicative of the speed of the vehicle, or
processing, using the neural network, the data indicative of the speed of the vehicle together with the audio feature.

17 . The one or more non-transitory computer-readable media of claim 15 , wherein:
the transducer is a first transducer disposed at a first position on the vehicle; the audio signal is a first audio signal; the operations comprise receiving a second audio signal from a second transducer disposed at a second position on the vehicle; and the estimating of the characteristic is further based on the second audio signal.

18 . The one or more non-transitory computer-readable media of claim 15 , wherein the operations comprise:
determining, based at least in part on the estimated characteristic, that the environment is outside of an approved ODD for the vehicle; and suspending full autonomous driving operations based at least in part on the modifying of the parameter.

19 . The one or more non-transitory computer-readable media of claim 15 , wherein the characteristic comprises a classification of precipitation in the environment of the vehicle.

20 . The one or more non-transitory computer-readable media of claim 15 , wherein the audio sensor is a microphone.

Assignees

• Zoox Inc

Inventors

• Zhong Xuan
• Ma Derek Xiang
• Terefe Nehemia Girma
• Ren Hang
• Subasingha Shaminda
• Logan David Donald

Classifications

• B60W60/0059

  Estimation of the risk associated with autonomous or manual driving, e.g. situation too complex, sensor failure or driver incapacity · CPC title

• B60W60/00182

  in response to weather conditions · CPC title

• B60W2050/0088

  Adaptive recalibration · CPC title

• B60W40/02 Primary

  related to ambient conditions · CPC title

• B60W2050/0057

  Frequency analysis, spectral techniques or transforms · CPC title