Emergency vehicle audio detection

Patent metadata

Field	Value
Publication number	US-11609576-B2
Application number	US-201916704171-A
Country	US
Kind code	B2
Filing date	Dec 5, 2019
Priority date	Dec 5, 2019
Publication date	Mar 21, 2023
Grant date	Mar 21, 2023

Abstract

In one embodiment, a process is performed during controlling Autonomous Driving Vehicle (ADV). Microphone signals sense sounds in an environment of the ADV. The microphone signals are combined and filtered to form an audio signal having the sounds sensed in the environment of the ADV. A neural network is applied to the audio signal to detect a presence of an audio signature of an emergency vehicle siren. If the siren is detected, a change in the audio signature to make a determination as to whether the emergency vehicle siren is a) moving towards the ADV, or b) not moving towards the ADV. The ADV can make a driving decision, such as slowing down, stopping, and/or steering to a side, based on if the emergency vehicle siren is moving towards the ADV.

First claim

The invention claimed is:

1. A computer-implemented method of operating an autonomous driving vehicle (ADV), method comprising:
capturing a stream of audio signals using one or more audio capturing devices mounted on the ADV, the stream of audio signals representing an audible environment surrounding the ADV;
storing the stream of audio signals in a buffer;
applying a neural network to at least a portion of the stream of audio signals received from the buffer, to detect an emergency vehicle siren signal;
in response to detecting the emergency vehicle siren signal by the neural network, performing an audio analysis on the stream of audio signals received from the buffer to determine whether an emergency vehicle is moving towards the ADV or moving away from the ADV, including in response to when the ADV is moving and an amplitude and a frequency of the emergency vehicle siren signal is constant, analyzing camera data to determine whether the emergency vehicle is moving towards the ADV or moving away from the ADV;
planning a trajectory to control the ADV based on determination of whether the emergency vehicle is moving towards or moving away from the ADV, including ignoring the emergency vehicle in response to determining that the emergency vehicle is moving away;
generating a control command that includes a throttle command, a brake command, or a steering command, based on the trajectory; and
applying the throttle command, the brake command, or the steering command to a throttle, a brake, or a steering unit of the ADV, respectively, resulting in a movement of the ADV according to the trajectory.

2. The method of claim 1 , further comprising controlling the ADV including at least one of steering the ADV out of a current driving lane or braking the ADV to decelerate, in response to determining that the emergency vehicle is moving towards the ADV.

3. The method of claim 1 , wherein performing the audio analysis on the at least a portion of the stream of audio signals comprises detecting a change in at least one of an amplitude or a frequency of an audio pattern across a plurality of audio frames of the audio signals.

4. The method of claim 3 , further comprising determining that the emergency vehicle siren signal is moving towards the ADV, in response to detecting an increase in the amplitude or the frequency.

5. The method of claim 1 , further comprising performing a digital signal processing (DSP) operation on the audio signals to remove noise.

6. The method of claim 1 , further comprising filtering the stream of audio signals to remove noise prior to storing the stream of audio signals as a plurality of audio frames in the buffer.

7. The method of claim 1 , wherein the neural network is a convolutional neural network that is trained with audio data representing emergency vehicle siren collected from a plurality of emergency vehicles.

8. The method of claim 1 , wherein performing the audio analysis on the at least a portion of the stream of audio signals to determine whether an emergency vehicle is moving towards the ADV or moving away from the ADV includes analyzing changes in an amplitude and frequency of an audio pattern across a plurality of audio frames of the audio signals with respect to a movement of the ADV.

9. A non-transitory machine-readable medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations of operating an autonomous driving vehicle (ADV), the operations comprising:
capturing a stream of audio signals using one or more audio capturing devices mounted on the ADV, the stream of audio signals representing an audible environment surrounding the ADV;
storing the stream of audio signals in a buffer;
applying a neural network to at least a portion of the stream of audio signals received from the buffer to detect an emergency vehicle siren signal;
in response to detecting the emergency vehicle siren signal, performing an audio analysis on the stream of audio signals received from buffer to determine whether an emergency vehicle is moving towards the ADV or moving away from the ADV, including in response to when the ADV is moving and an amplitude and a frequency of the emergency vehicle siren signal is constant, analyzing camera data to determine whether the emergency vehicle is moving towards the ADV or moving away from the ADV;
planning a trajectory to control the ADV based on determination of whether the emergency vehicle is moving towards or moving away from the ADV, including ignoring the emergency vehicle in response to determining that the emergency vehicle is moving away;
generating a control command that includes a throttle command, a brake command, or a steering command based on the trajectory; and
applying the throttle command, the brake command, or the steering command to a throttle, a brake, or a steering unit of the ADV, respectively, resulting in a movement of the ADV according to the trajectory.

10. The machine-readable medium of claim 9 , wherein the operations further comprise controlling the ADV including at least one of steering the ADV out of a current driving lane or braking the ADV to decelerate, in response to determining that the emergency vehicle is moving towards the ADV.

11. The machine-readable medium of claim 9 , wherein performing the audio analysis on the at least a portion of the stream of audio signals comprises detecting a change in at least one of an amplitude or a frequency of an audio pattern across a plurality of audio frames of the audio signals.

12. The machine-readable medium of claim 11 , wherein the operations further comprise determining that the emergency vehicle siren signal is moving towards the ADV, in response to detecting an increase in the amplitude or the frequency.

13. The machine-readable medium of claim 9 , wherein the operations further comprise performing a digital signal processing (DSP) operation on the audio signals to remove noise.

14. The machine-readable medium of claim 9 , further comprising filtering the stream of audio signals to remove noise prior to storing the stream of audio signals as a plurality of audio frames in the buffer.

15. The machine-readable medium of claim 9 , wherein the neural network is a convolutional neural network that is trained with audio data representing emergency vehicle siren collected from a plurality of emergency vehicles.

16. The machine-readable medium of claim 9 , wherein performing the audio analysis on the at least a portion of the stream of audio signals to determine whether an emergency vehicle is moving towards the ADV or moving away from the ADV includes analyzing changes in an amplitude and frequency of an audio pattern across a plurality of audio frames of the audio signals with respect to a movement of the ADV.

17. A data processing system, comprising:
a processor; and
a memory coupled to the processor to store instructions, which when executed by the processor, cause the processor to perform operations of operating an autonomous driving vehicle (ADV), the operations comprising:
capturing a stream of audio signals using one or more audio capturing devices mounted on the ADV, the stream of audio signals representing an audible environment surrounding the ADV,
storing the stream of audio signals in a buffer;
applying a neural network to at least a portion of the stream of audio signals received from the buffer, to detect an emergency vehicle siren signal,
in response to detecting the emergency vehicle siren signal by the neural network, performing an audio analysis on the stream of audio signa

Assignees

• Baidu Usa Llc

Inventors

• Luo Qi
• Xu Kecheng
• Zhou Jinyun
• Xiao Xiangquan
• Huang Shuo
• Hu Jiangtao
• Miao Jinghao

Classifications

• G10L25/51

  for comparison or discrimination · CPC title

• B60W30/09

  Taking automatic action to avoid collision, e.g. braking and steering · CPC title

• G10L25/30

  using neural networks · CPC title

• B60W60/0027

  using trajectory prediction for other traffic participants · CPC title

• B60W2420/60

  Doppler effect · CPC title