Method and apparatus with abnormal channel of microphone array detection and compensation signal generation

Patent metadata

Field	Value
Publication number	US-2023104123-A1
Application number	US-202217689299-A
Country	US
Kind code	A1
Filing date	Mar 8, 2022
Priority date	Oct 6, 2021
Publication date	Apr 6, 2023
Grant date	—

Abstract

A method includes: receiving multi-channel sound source signals from a microphone array; synchronizing the multi-channel sound source signals based on spatial information of the microphone array; and detecting an abnormal channel of the microphone array by inputting the synchronized sound source signals and first conditional information to a neural network model configured to perform an inverse operation.

First claim

What is claimed is:

1 . A method, comprising:
receiving multi-channel sound source signals from a microphone array; synchronizing the multi-channel sound source signals based on spatial information of the microphone array; and detecting an abnormal channel of the microphone array by inputting the synchronized sound source signals and first conditional information to a neural network model configured to perform an inverse operation.

2 . The method of claim 1 , further comprising:
generating a compensation signal corresponding to the abnormal channel using the neural network model in response to the abnormal channel being detected.

3 . The method of claim 1 , further comprising:
determining a sound source signal of a reference channel of the microphone array from among the multi-channel sound source signals; and determining the sound source signal of the reference channel to be the first conditional information.

4 . The method of claim 3 , wherein the synchronizing further comprises shifting the multi-channel sound source signals based on the sound source signal of the reference channel.

5 . The method of claim 1 , wherein the detecting of the abnormal channel comprises:
determining an output vector by inputting the synchronized sound source signals and the first conditional information to the neural network model; determining a probability value corresponding to the output vector; and detecting the abnormal channel of the microphone array by comparing the probability value to a threshold.

6 . The method of claim 1 , wherein the spatial information of the microphone array comprises either one or both of shape information of the microphone array and distance information between channels included in the microphone array.

7 . The method of claim 2 , wherein the generating of the compensation signal comprises:
sampling an arbitrary vector in a probability distribution corresponding to an output of the neural network model; generating an intermediate compensation signal by inputting the arbitrary vector and second conditional information to the neural network model; and generating, as the compensation signal, a final compensation signal by shifting the intermediate compensation signal based on the spatial information of the microphone array.

8 . The method of claim 7 , wherein the second conditional information comprises a sound source signal corresponding to one of the channels other than the abnormal channel.

9 . A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, configure the one or more processors to perform the method of claim 1 .

10 . An apparatus, comprising:
one or more processor configured to:
receive multi-channel sound source signals from a microphone array;
synchronize the multi-channel sound source signals based on spatial information of the microphone array; and
detect an abnormal channel of the microphone array by inputting the synchronized sound source signals and first conditional information to a neural network model configured to perform an inverse operation.

11 . The apparatus of claim 10 , wherein the one or more processors are further configured to generate a compensation signal corresponding to the abnormal channel using the neural network model in response to the abnormal channel being detected.

12 . The apparatus of claim 10 , wherein the one or more processors are further configured to:
determine a sound source signal of a reference channel of the microphone array from among the multi-channel sound source signals; and determine the sound source signal of the reference channel to be the first conditional information.

13 . The apparatus of claim 12 , wherein, for the synchronizing, the one or more processors are further configured to shift the multi-channel sound source signals based on the sound source signal of the reference channel.

14 . The apparatus of claim 10 , wherein, for the detecting of the abnormal channel, the one or more processors are further configured to:
determine an output vector by inputting the synchronized sound source signals and the first conditional information to the neural network model; determine a probability value corresponding to the output vector; and detect the abnormal channel of the microphone array by comparing the probability value to a threshold.

15 . The apparatus of claim 10 , wherein the spatial information of the microphone array comprises either one or both of shape information of the microphone array and distance information between channels included in the microphone array.

16 . The apparatus of claim 11 , wherein, for the generating of the compensation signal, the one or more processors are further configured to:
sample an arbitrary vector in a probability distribution corresponding to an output of the neural network model; generate an intermediate compensation signal by inputting the arbitrary vector and second conditional information to the neural network model; and generate, as the compensation signal, a final compensation signal by shifting the intermediate compensation signal based on the spatial information of the microphone array.

17 . The apparatus of claim 16 , wherein the second conditional information comprises a sound source signal corresponding to one of the channels other than the abnormal channel.

18 . An electronic device, comprising:
a microphone array configured to receive multi-channel sound source signals; and one or more processors configured to:
synchronize the multi-channel sound source signals based on spatial information of the microphone array; and
detect an abnormal channel of the microphone array by inputting the synchronized sound source signals and first conditional information to a neural network model configured to perform an inverse operation.

19 . The electronic device of claim 18 , wherein the one or more processors are further configured to generate a compensation signal corresponding to the abnormal channel using the neural network model in response to the abnormal channel being detected.

20 . A method, comprising:
sampling an arbitrary vector in a probability distribution corresponding to an output of a neural network model configured to perform an inverse operation; generating an intermediate compensation signal by inputting the arbitrary vector and second conditional information to the neural network model; and generating a compensation signal corresponding to a detected abnormal channel of a microphone array by shifting the intermediate compensation signal based on spatial information of the microphone array.

21 . The method of claim 20 , further comprising:
receiving multi-channel sound source signals from the microphone array; synchronizing the multi-channel sound source signals based on the spatial information; and detecting the abnormal channel by inputting the synchronized sound source signals and first conditional information to the neural network model.

22 . The method of claim 21 , wherein the synchronizing further comprises:
determining delay times among the channels of the microphone array based on the spatial information of the microphone array; and synchronizing the multi-channel sound source signals by shifting the multi-channel sound source signals based on the delay times.

23 . The method of claim 20 , wherein the spatial information of the microphone array comprises info

Assignees

• Samsung Electronics Co Ltd
• Seoul Nat Univ R&Db Foundation

Inventors

• Kim Nam Soo
• Yoon Ji Won

Classifications

• H04R29/005 Primary

  Microphone arrays · CPC title

• H04R3/005

  for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407) · CPC title

• H04R5/04

  Circuit arrangements, {e.g. for selective connection of amplifier inputs/outputs to loudspeakers, for loudspeaker detection, or for adaptation of settings to personal preferences or hearing impairments (combinations of amplifiers H03F3/68; stereophonic systems H04S)} · CPC title

• H04R2201/401

  2D or 3D arrays of transducers · CPC title

• H04R1/406

  microphones · CPC title