Multi-channel voice activity detection

Patent metadata

Field	Value
Publication number	US-12154547-B2
Application number	US-202318471627-A
Country	US
Kind code	B2
Filing date	Sep 21, 2023
Priority date	Oct 22, 2020
Publication date	Nov 26, 2024
Grant date	Nov 26, 2024

Abstract

A method for multi-channel voice activity detection includes receiving a sequence of input frames characterizing streaming multi-channel audio captured by an array of microphones. Each channel of the streaming multi-channel audio includes respective audio features captured by a separate dedicated microphone. The method also includes determining, using a location fingerprint model, a location fingerprint indicating a location of a source of the multi-channel audio relative to the user device based on the respective audio features of each channel of the multi-channel audio. The method also includes generating an output from an application-specific classifier. The first score indicates a likelihood that the multi-channel audio corresponds to a particular audio type that the particular application is configured to process. The method also includes determining whether to accept or reject the multi-channel audio for processing by the particular application based on the first score generated as output from the application-specific classifier.

First claim

What is claimed is:

1. A computer-implemented method when executed on data processing hardware of a user device causes the data processing hardware to perform operations comprising:
receiving streaming multi-channel audio captured by an array of microphones in communication with the data processing hardware, each channel of the multi-channel audio comprising respective audio features captured by a separate dedicated microphone in the array of microphones;
processing the respective audio features of each channel of the multi-channel audio to determine an embedding associated with a source of the multi-channel audio;
based on the embedding associated with the source of the multi-channel audio, determining a first score indicating the multi-channel audio originates from one of a moving source or a static source; and
determining to reject the multi-channel audio for processing by a particular application based on the first score indicating that the multi-channel audio originates from the moving source.

2. The computer-implemented method of claim 1 , wherein the embedding associated with the source of the multi-channel audio comprises a location embedding indicating a location of the source of the multi-channel audio relative to the user device.

3. The computer-implemented method of claim 1 , wherein the embedding associated with the source of the multi-channel audio comprises a direction embedding indicating a direction of the source of the multi-channel audio relative to the user device.

4. The computer-implemented method of claim 1 , wherein determining the first score indicating that the multi-channel audio originates from one of the moving source or the static source comprises executing a classifier model configured to:
receive, as input, the embedding associated with the multi-channel audio; and
generate, as output, the first score indicating a likelihood that the multi-channel audio originates from one of the moving source or the static source.

5. The computer-implemented method of claim 1 , wherein the operations further comprise determining that the particular application is configured to process static source audio.

6. The computer-implemented method of claim 5 , wherein the operations further comprise:
determining that the first score satisfies a first score threshold; and
based on determining that the first score satisfies the first score threshold, rejecting the multi-channel audio for processing by the particular application.

7. The computer-implemented method of claim 5 , wherein the operations further comprise determining, using a voice activity detector (VAD) model, a second score indicating a likelihood that the multi-channel audio corresponds to human-originated speech.

8. The computer-implemented method of claim 7 , wherein the operations further comprise:
combining the first score and the second score into a combined score;
determining that the combined score fails to satisfy an acceptance threshold; and
based on determining that the combined score fails to satisfy the acceptance threshold, rejecting the multi-channel audio for processing by the particular application.

9. The computer-implemented method of claim 1 , wherein processing the respective audio features of each channel of the multi-channel audio to determine the embedding associated with a source of the multi-channel audio comprises processing each channel of the multi-channel audio using a time difference of arrival and gain model.

10. The computer-implemented method of claim 1 , wherein processing the respective audio features of each channel of the multi-channel audio to determine the embedding associated with a source of the multi-channel audio comprises processing each channel of the multi-channel audio using a spatial probability model.

11. A system comprising:
data processing hardware of a user device; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
receiving streaming multi-channel audio captured by an array of microphones in communication with the data processing hardware, each channel of the multi-channel audio comprising respective audio features captured by a separate dedicated microphone in the array of microphones;
processing the respective audio features of each channel of the multi-channel audio to determine an embedding associated with a source of the multi-channel audio;
based on the embedding associated with the source of the multi-channel audio, determining a first score indicating the multi-channel audio originates from one of a moving source or a static source; and
determining to reject the multi-channel audio for processing by a particular application based on the first score indicating that the multi-channel audio originates from the moving source.

12. The system of claim 11 , wherein the embedding associated with the source of the multi-channel audio comprises a location embedding indicating a location of the source of the multi-channel audio relative to the user device.

13. The system of claim 11 , wherein the embedding associated with the source of the multi-channel audio comprises a direction embedding indicating a direction of the source of the multi-channel audio relative to the user device.

14. The system of claim 11 , wherein determining the first score indicating that the multi-channel audio originates from one of the moving source or the static source comprises executing a classifier model configured to:
receive, as input, the embedding associated with the multi-channel audio; and
generate, as output, the first score indicating a likelihood that the multi-channel audio originates from one of the moving source or the static source.

15. The system of claim 11 , wherein the operations further comprise determining that the particular application is configured to process static source audio.

16. The system of claim 11 , wherein the operations further comprise:
determining that the first score fails to satisfy a first score threshold; and
based on determining that the first score fails to satisfy the first score threshold, rejecting the multi-channel audio for processing by the particular application.

17. The system of claim 11 , wherein the operations further comprise determining, using a voice activity detector (VAD) model, a second score indicating a likelihood that the multi-channel audio corresponds to human-originated speech.

18. The system of claim 17 , wherein the operations further comprise:
combining the first score and the second score into a combined score;
determining that the combined score fails to satisfy an acceptance threshold; and
based on determining that the combined score fails to satisfy the acceptance threshold, rejecting the multi-channel audio for processing by the particular application.

19. The system of claim 11 , wherein processing the respective audio features of each channel of the multi-channel audio to determine the embedding associated with a source of the multi-channel audio comprises processing each channel of the multi-channel audio using a time difference of arrival and gain model.

20. The system of claim 11 , wherein processing the respective audio features of each channel of the multi-channel audio to determine the embedding associated with a source of the multi-channel audio comprises processing each channel of the multi-channel audio using a spatial probability

Assignees

• Google Llc

Inventors

• Miller Nolan Andrew
• Mehran Ramin

Classifications

• H04R3/005

  for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407) · CPC title

• G10L2021/02166

  Microphone arrays; Beamforming · CPC title

• G10L15/02 Primary

  Feature extraction for speech recognition; Selection of recognition unit · CPC title

• G10L25/51

  for comparison or discrimination · CPC title

• G10L21/0216

  characterised by the method used for estimating noise · CPC title