End-to-end deep neural network for auditory attention decoding

Patent metadata

Field	Value
Publication number	US-11630513-B2
Application number	US-201916720810-A
Country	US
Kind code	B2
Filing date	Dec 19, 2019
Priority date	Dec 20, 2018
Publication date	Apr 18, 2023
Grant date	Apr 18, 2023

Abstract

In one aspect of the present disclosure, method includes: receiving neural data responsive to a listener's auditory attention; receiving an acoustic signal responsive to a plurality of acoustic sources; for each of the plurality of acoustic sources: generating, from the received acoustic signal, audio data comprising one or more features of the acoustic source, forming combined data representative of the neural data and the audio data, and providing the combined data to a classification network configured to calculate a similarity score between the neural data and the acoustic source using one or more similarity metrics; and using the similarity scores calculated for each of the acoustic sources to identify, from the plurality of acoustic sources, an acoustic source associated with the listener's auditory attention.

First claim

The invention claimed is:

1. A method comprising:
receiving neural data responsive to a listener's auditory attention;
receiving an acoustic signal responsive to a plurality of acoustic sources;
for each of the plurality of acoustic sources:
generating, from the received acoustic signal, audio data comprising one or more features of the acoustic source,
forming combined data representative of the neural data and the audio data, and
providing the combined data to a convolutional deep neural network (DNN) configured to calculate a similarity score between the neural data and the acoustic source using one or more similarity metrics; and

using the similarity scores calculated for each of the acoustic sources to identify, from the plurality of acoustic sources, an acoustic source associated with the listener's auditory attention.

2. The method of claim 1 , comprising separating the acoustic signal into a plurality of candidate signals, wherein generating audio data for an acoustic source comprises generating audio data using one of the plurality of candidate signals associated with the acoustic source.

3. The method of claim 1 , comprising:
receiving one or more neural signals responsive to brain activity of the listener; and
processing the one or more neural signals to generate the neural data.

4. The method of claim 3 , wherein a device worn by the listener receives the one or more neural signals is and a companion device calculates the similarity scores.

5. The method of claim 1 , wherein the neural data comprises multi-channel electroencephalogram (EEG) data.

6. The method of claim 1 , wherein forming the combined data comprises generating a matrix comprising the neural data and the audio data.

7. The method of claim 1 , wherein the convolutional DNN comprises at least two convolutional layers and at least three fully connected layers.

8. The method of claim 1 , wherein using the similarity scores calculated for each of the acoustic sources to identify the acoustic source associated with the listener's auditory attention comprises identifying an acoustic source having the highest similarity score.

9. An apparatus comprising:
a neural sensor interface configured to receive one or more neural signals responsive to a listener's auditory attention;
an audio input configured to receive an acoustic signal responsive to a plurality of acoustic sources; and
a processor configured to:
process the one or more neural signals to generate multi-channel neural data;
for each of the plurality of acoustic sources:
generate, from the received acoustic signal, audio data comprising one or more features of the acoustic source,
form combined data representative of the neural data and the audio data, and
provide the combined data to a convolutional deep neural network (DNN) configured to calculate a similarity score between the neural data and the acoustic source using one or more similarity metrics; and
identify, from the plurality of acoustic sources, an acoustic source associated with the listener's auditory attention based on the calculated similarity scores.

10. The apparatus of claim 9 , wherein the processor is configured to separate the acoustic signal into a plurality of candidate signals, and to generate audio data for an acoustic source using one of the plurality of candidate signals associated with the acoustic source.

11. The apparatus of claim 10 , comprising an audio output, the processor configured to provide, to the audio output, a candidate signal from the plurality of candidate signals corresponding to the acoustic source associated with the listener's auditory attention.

12. The apparatus of claim 9 , wherein the neural sensor interface is configured to receive multi-channel electroencephalogram (EEG) measurements.

13. The apparatus of claim 9 , wherein the processor is configured to form the combined data comprises as a matrix comprising the neural data and the audio data.

14. The apparatus of claim 9 , wherein the convolutional DNN comprises at least two convolutional layers and at least three fully connected layers.

15. The apparatus of claim 9 , comprising a non-volatile memory configured to store the DNN as a trained model.

16. The apparatus of claim 9 , wherein the processor is configured to identify the acoustic source associated with the listener's auditory attention by identifying an acoustic source having the highest similarity score.

17. A non-transitory computer-readable medium storing program instructions that are executable to:
receive neural data responsive to a listener's auditory attention;
receive an acoustic signal responsive to a plurality of acoustic sources;
for each of the plurality of acoustic sources:
generate, from the received acoustic signal, audio data comprising one or more features of the acoustic source,
form combined data representative of the neural data and the audio data, and
provide the combined data to a convolutional deep neural network (DNN) configured to calculate a similarity score between the neural data and the acoustic source using one or more similarity metrics; and

use the similarity scores calculated for each of the acoustic sources to identify, from the plurality of acoustic sources, an acoustic source associated with the listener's auditory attention.

Assignees

• Massachusetts Inst Technology
• Univ Columbia

Inventors

• Ciccarelli Gregory
• Smalt Christopher
• Quatieri Thomas
• Brandstein Michael
• Calamia Paul
• Haro Stephanie
• Nolan Michael
• Perricone Joseph
• Mesgarani Nima
• O'Sullivan James

Classifications

• G06N3/09

  Supervised learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06F18/22

  Matching criteria, e.g. proximity measures · CPC title

• A61B5/121

  evaluating hearing capacity · CPC title

• G06F3/017

  Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883) · CPC title