Processing of audio signals from multiple microphones

Patent metadata

Field	Value
Publication number	US-2023036986-A1
Application number	US-202217814660-A
Country	US
Kind code	A1
Filing date	Jul 25, 2022
Priority date	Jul 27, 2021
Publication date	Feb 2, 2023
Grant date	—

Abstract

A first device includes a memory configured to store instructions and one or more processors configured to receive audio signals from multiple microphones. The one or more processors are configured to process the audio signals to generate direction-of-arrival information corresponding to one or more sources of sound represented in one or more of the audio signals. The one or more processors are also configured to and send, to a second device, data based on the direction-of-arrival information and a class or embedding associated with the direction-of-arrival information.

First claim

What is claimed is:

1 . A first device comprising:
a memory configured to store instructions; and one or more processors configured to:
receive audio signals from multiple microphones;
process the audio signals to generate direction-of-arrival information corresponding to one or more sources of sound represented in one or more of the audio signals; and
send, to a second device, data based on the direction-of-arrival information and a class or embedding associated with the direction-of-arrival information.

2 . The first device of claim 1 , wherein the one or more processors are further configured to process signal data corresponding to the audio signals to determine the class or embedding.

3 . The first device of claim 2 , wherein the one or more processors are further configured to perform a beamforming operation on the audio signals to generate the signal data.

4 . The first device of claim 2 , wherein the one or more processors are further configured to process the signal data at one or more classifiers to determine the class from among multiple classes supported by the one or more classifiers, for a sound represented in one or more of the audio signals and associated with an audio event, and wherein the class is sent to the second device.

5 . The first device of claim 2 , wherein the one or more processors are further configured to process the signal data at one or more encoders to generate the embedding, the embedding corresponding to a sound represented in one or more of the audio signals and associated with an audio event, and wherein the embedding is sent to the second device.

6 . The first device of claim 1 , wherein one or more processors are further configured to process image data at one or more encoders to generate the embedding, the embedding corresponding to an object represented in the image data and associated with an audio event, and wherein the embedding is sent to the second device.

7 . The first device of claim 6 , further comprising one or more cameras configured to generate the image data.

8 . The first device of claim 1 , wherein:
the class corresponds to a category for a particular sound represented in the audio signals and associated with a particular audio event; and the embedding includes a signature or information that corresponds to the particular sound or the particular audio event and is configured to enable detection, via processing of other audio signals, of the particular sound or the particular audio event in the other audio signals.

9 . The first device of claim 1 , wherein the one or more processors are further configured to:
perform spatial processing on the audio signals based on the direction-of-arrival information to generate one or more beamformed audio signals; and send the one or more beamformed audio signals to the second device.

10 . The first device of claim 1 , wherein the memory and the one or more processors are integrated into a headset device, and wherein the second device corresponds to a mobile phone.

11 . The first device of claim 1 , further comprising a modem, wherein the data is sent to the second device via the modem.

12 . The first device of claim 1 , wherein the one or more processors are further configured to send a representation of the audio signals to the second device.

13 . The first device of claim 12 , wherein the representation of the audio signals corresponds to one or more beamformed audio signals.

14 . The first device of claim 1 , wherein the one or more processors are further configured to generate a user interface output indicative of at least one of an environmental event or an acoustic event.

15 . The first device of claim 1 , wherein the one or more processors are further configured to receive, from the second device, data indicative of an acoustic event.

16 . The first device of claim 1 , wherein the one or more processors are further configured to:
receive, from the second device, directional information associated with the audio signals; and perform an audio zoom operation based on the directional information.

17 . The first device of claim 1 , wherein the one or more processors are integrated in a vehicle.

18 . The first device of claim 1 , wherein the data based on the direction-of-arrival information includes a report indicating at least one detected event and a direction of the detected event.

19 . The first device of claim 1 , further comprising the multiple microphones.

20 . The first device of claim 1 , further comprising at least one speaker configured to output a sound associated with at least one of the audio signals.

21 . A method of processing audio, the method comprising:
receiving, at one or more processors of a first device, audio signals from multiple microphones; processing the audio signals to generate direction-of-arrival information corresponding to one or more sources of sound represented in one or more of the audio signals; and sending, to a second device, data based on the direction-of-arrival information and a class or embedding associated with the direction-of-arrival information.

22 . The method of claim 21 , further comprising processing signal data corresponding to the audio signals to determine the class or embedding.

23 . The method of claim 22 , further comprising performing a beamforming operation on the audio signals to generate the signal data.

24 . The method of claim 22 , wherein the signal data is processed at one or more classifiers to determine the class from among multiple classes supported by the one or more classifiers, for a sound represented in one or more of the audio signals and associated with an audio event, and wherein the class is sent to the second device.

25 . The method of claim 22 , wherein the signal data is processed at one or more encoders to generate the embedding, the embedding corresponding to a sound represented in one or more of the audio signals and associated with an audio event, and wherein the embedding is sent to the second device.

26 . The method of claim 21 , further comprising sending a representation of the audio signals to the second device.

27 . The method of claim 21 , further comprising:
receiving, at one or more processors of the second device, the data based on the direction-of-arrival information and the class; obtaining, at the one or more processors of the second device, audio data representing a sound associated with the direction-of-arrival information and the class; and verifying the class, at the one or more processors of the second device, based on at least the audio data and the direction-of-arrival information.

28 . The method of claim 21 , further comprising:
receiving, at one or more processors of the second device, the data based on the direction-of-arrival information and the embedding; and processing, at the one or more processors of the second device, audio data representing a sound scene based on the direction-of-arrival information and the embedding to generate modified audio data corresponding to an updated sound scene.

29 . A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of a first device, cause the one or more processors to:
receive audio signals from multiple microphones; process the audio signals to generate direct

Assignees

• Qualcomm Inc

Inventors

• Visser Erik
• Saki Fatemeh
• Guo Yinyi
• Kim Lae-Hoon
• Alves Rogerio Guedes
• Pessentheiner Hannes

Classifications

• G06F3/165

  Management of the audio stream, e.g. setting of volume, audio stream path · CPC title

• G10L25/51

  for comparison or discrimination · CPC title

• H04R2460/01

  Hearing devices using active noise cancellation · CPC title

• G10L2021/02166

  Microphone arrays; Beamforming · CPC title

• H04R2430/23

  Direction finding using a sum-delay beam-former · CPC title