Multi channel voice activity detection

Patent metadata

Field	Value
Publication number	US-11380302-B2
Application number	US-202017077679-A
Country	US
Kind code	B2
Filing date	Oct 22, 2020
Priority date	Oct 22, 2020
Publication date	Jul 5, 2022
Grant date	Jul 5, 2022

Abstract

A method for multi-channel voice activity detection includes receiving a sequence of input frames characterizing streaming multi-channel audio captured by an array of microphones. Each channel of the streaming multi-channel audio includes respective audio features captured by a separate dedicated microphone. The method also includes determining, using a location fingerprint model, a location fingerprint indicating a location of a source of the multi-channel audio relative to the user device based on the respective audio features of each channel of the multi-channel audio. The method also includes generating an output from an application-specific classifier. The first score indicates a likelihood that the multi-channel audio corresponds to a particular audio type that the particular application is configured to process. The method also includes determining whether to accept or reject the multi-channel audio for processing by the particular application based on the first score generated as output from the application-specific classifier.

First claim

What is claimed is:

1. A method comprising:
receiving, at data processing hardware of a user device, a sequence of input frames characterizing streaming multi-channel audio captured by an array of microphones in communication with the data processing hardware, each channel of the streaming multi-channel audio comprising respective audio features captured by a separate dedicated microphone in the array of microphones;
determining, by the data processing hardware, using a location fingerprint model, a location fingerprint indicating a location of a source of the multi-channel audio relative to the user device based on the respective audio features of each channel of the multi-channel audio;
generating, by the data processing hardware, as output from an application-specific classifier configured to receive the location fingerprint as input, a first score indicating a likelihood that the multi-channel audio corresponds to a particular audio type that a particular application is configured to process; and
determining, by the data processing hardware, whether to accept or reject the multi-channel audio for processing by the particular application based on the first score generated as output from the application-specific classifier.

2. The method of claim 1 , further comprising:
generating, by the data processing hardware, using a voice activity detector (VAD) model, a second score indicating a likelihood that the multi-channel audio corresponds to human-originated speech,
wherein determining whether to accept or reject the multi-channel audio for processing by the particular application is further based on the second score indicating the likelihood that the multi-channel audio corresponds to human-originated speech.

3. The method of claim 2 , wherein determining whether to accept or reject the multi-channel audio for processing by the particular application comprises:
combining the first score and the second score into a combined score;
determining whether the combined score satisfies an acceptance threshold; and one of:
when the combined score satisfies the acceptance threshold, accepting the multi-channel audio for processing by the particular application; or
when the combined score fails to satisfy the acceptance threshold, rejecting the multi-channel audio for processing by the particular application.

4. The method of claim 2 , further comprising:
generating, by the data processing hardware, an aggregated fingerprint based on the location fingerprint and one or more previous location fingerprints;
extracting, by the data processing hardware, using a beamformer configured to receive the aggregated fingerprint as input, a single channel of audio data from the multi-channel audio, the extracted single channel of audio data including only respective audio features that correspond to the location of the source indicated by the location fingerprint, and
wherein generating the second score indicating the likelihood that the multi-channel audio corresponds to human-originated speech comprises generating the second score as output from the VAD model based on the extracted single channel of audio data received as input to the VAD model.

5. The method of claim 4 , further comprising adjusting, by the data processing hardware, the second score based on a confidence level of the beamformer.

6. The method of claim 1 , wherein the particular audio type that the particular application is configured to process comprises one of audio with a single source location or audio with a multiple source location.

7. The method of claim 1 , wherein the particular audio type that the particular application is configured to process comprises one of audio with a moving source location or audio with a static source location.

8. The method of claim 1 , wherein the particular audio type that the particular application is configured to process comprises one of near source audio or far source audio.

9. The method of claim 1 , wherein the particular audio type that the particular application is configured to process comprises one of point source audio or speaker system audio.

10. The method of claim 1 , wherein determining the location fingerprint indicating the location of the source of the multi-channel audio relative to the user device comprises processing each channel of the multi-channel audio using a time difference of arrival and gain model.

11. The method of claim 1 , wherein determining the location fingerprint indicating the location of the source of the multi-channel audio relative to the user device comprises processing each channel of the multi-channel audio using a spatial probability model.

12. The method of claim 1 , further comprising:
generating, by the data processing hardware, as output from the application-specific classifier, based on the location fingerprint, a second score indicating a likelihood that the multi-channel audio corresponds to an audio type different than the particular audio type that the particular application is configured to process; and
ignoring, by the data processing hardware, subsequent streaming multi-channel audio with the same location fingerprint.

13. The method of claim 1 , wherein the application-specific classifier is trained on:
positive training samples comprising multi-channel audio corresponding to the particular audio type that the particular application is configured to process; and
negative training samples comprising multi-channel audio corresponding to one or more other audio types that the particular application is not configured to process.

14. A system comprising:
data processing hardware of a user device; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
receiving, a sequence of input frames characterizing streaming multi-channel audio captured by an array of microphones in communication with the data processing hardware, each channel of the streaming multi-channel audio comprising respective audio features captured by a separate dedicated microphone in the array of microphones;
determining, using a location fingerprint model, a location fingerprint indicating a location of a source of the multi-channel audio relative to the user device based on the respective audio features of each channel of the multi-channel audio;
generating, as output from an application-specific classifier configured to receive the location fingerprint as input, a first score indicating a likelihood that the multi-channel audio corresponds to a particular audio type that a particular application is configured to process; and
determining whether to accept or reject the multi-channel audio for processing by the particular application based on the first score generated as output from the application-specific classifier.

15. The system of claim 14 , wherein the operations further comprise:
generating, using a voice activity detector (VAD) model, a second score indicating a likelihood that the multi-channel audio corresponds to human-originated speech,
wherein determining whether to accept or reject the multi-channel audio for processing by the particular application is further based on the second score indicating the likelihood that the multi-channel audio corresponds to human-originated speech.

16. The system of claim 15 , wherein determining whether to accept or reject the multi-channel audio for processing by the particular application comprises:
combining the first score and the second sco

Assignees

• Google Llc

Inventors

• Miller Nolan Andrew
• Mehran Ramin

Classifications

• H04R3/005

  for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407) · CPC title

• G10L15/02 Primary

  Feature extraction for speech recognition; Selection of recognition unit · CPC title

• G10L2021/02166

  Microphone arrays; Beamforming · CPC title

• G10L25/78 Primary

  Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10) · CPC title

• G10L25/51

  for comparison or discrimination · CPC title