Conference segmentation based on conversational dynamics

Patent metadata

Field	Value
Publication number	US-2018336902-A1
Application number	US-201615546109-A
Country	US
Kind code	A1
Filing date	Feb 3, 2016
Priority date	Feb 3, 2015
Publication date	Nov 22, 2018
Grant date	—

Abstract

Various disclosed implementations involve processing and/or playback of a recording of a conference involving a plurality of conference participants. Some implementations disclosed herein involve analyzing conversational dynamics of the conference recording. Some examples may involve searching the conference recording to determine instances of segment classifications. The segment classifications may be based, at least in part, on conversational dynamics data. Some implementations may involve segmenting the conference recording into a plurality of segments, each of the segments corresponding with a time interval and at least one of the segment classifications. Some implementations allow a listener to scan through a conference recording quickly according to segments, words, topics and/or talkers of interest.

First claim

What is claimed is:

1 . A method for processing audio data, the method comprising:
receiving, by a conversational dynamics analysis module, audio data corresponding to a conference recording of a conference involving a plurality of conference participants, the audio data including at least one of: (a) conference participant speech data from multiple endpoints, recorded separately or (b) conference participant speech data from a single endpoint corresponding to multiple conference participants and including information for identifying conference participant speech for each conference participant of the multiple conference participants; analyzing conversational dynamics of the conference recording to determine conversational dynamics data; searching the conference recording to determine instances of each of a plurality of segment classifications, each of the segment classifications based, at least in part, on the conversational dynamics data; and segmenting the conference recording into a plurality of segments, each of the segments corresponding with a time interval and at least one of the segment classifications, wherein the analyzing, searching and segmenting processes are performed by the conversational dynamics analysis module.

2 . The method of claim 1 , wherein the searching and segmenting processes are recursive.

3 . The method of claim 1 , wherein the searching and segmenting processes are performed multiple times at different time scales.

4 . The method of claim 1 , wherein the searching and segmenting processes are based, at least in part, on a hierarchy of segment classifications.

5 . The method of claim 4 , wherein the hierarchy of segment classifications is based upon one or more criteria from a list of criteria consisting of: a level of confidence with which segments of a particular segment classification may be identified; a level of confidence with which a start time of a segment may be determined; a level of confidence with which an end time of a segment may be determined; and a likelihood that a particular segment classification includes conference participant speech corresponding to a conference topic.

6 . The method of claim 1 , wherein instances of the segment classifications are determined according to a set of rules and wherein the rules are based on one or more conversational dynamics data types selected from a group of conversational dynamics data types consisting of: (a) a doubletalk ratio indicating a fraction of speech time in a time interval during which at least two conference participants are speaking simultaneously; (b) a speech density metric indicating a fraction of the time interval during which there is any conference participant speech; and (c) a dominance metric indicating a fraction of total speech uttered by a dominant conference participant during the time interval, the dominant conference participant being a conference participant who spoke the most during the time interval.

7 . The method of claim 6 , wherein the set of rules includes a rule that classifies a segment as a Mutual Silence segment if the speech density metric is less than a mutual silence threshold.

8 . The method of claim 7 , wherein the set of rules includes a rule that classifies a segment as a Babble segment if the speech density metric is greater than or equal to the mutual silence threshold and the doubletalk ratio is greater than a babble threshold.

9 . The method of claim 8 , wherein the set of rules includes a rule that classifies a segment as a Discussion segment if the speech density metric is greater than or equal to the silence threshold and if the doubletalk ratio is less than or equal to the babble threshold but greater than a discussion threshold.

10 . The method of claim 9 , wherein the set of rules includes a rule that classifies a segment as a Presentation segment if the speech density metric is greater than or equal to the silence threshold, if the doubletalk ratio is less than or equal to the discussion threshold and if the dominance metric is greater than a presentation threshold.

11 . The method of claim 10 , wherein the set of rules includes a rule that classifies a segment as a Question and Answer segment if the speech density metric is greater than or equal to the silence threshold, if the doubletalk ratio is less than or equal to the discussion threshold and if the dominance metric is less than or equal to the presentation threshold but greater than a question and answer threshold.

12 . The method of claim 11 , wherein the searching and segmenting processes are based, at least in part, on a hierarchy of segment classifications and wherein a first hierarchical level of the searching process involves searching the conference recording to determine instances of Babble segments.

13 . The method of claim 12 , wherein a second hierarchical level of the searching process involves searching the conference recording to determine instances of Presentation segments.

14 . The method of claim 13 , wherein a third hierarchical level of the searching process involves searching the conference recording to determine instances of Question and Answer segments and wherein a fourth hierarchical level of the searching process involves searching the conference recording to determine instances of Discussion segments.

15 . The method of claim 1 , wherein instances of the segment classifications are determined according to a machine learning classifier.

16 . The method of claim 15 , wherein the machine learning classifier is selected from a group of machine learning classifiers consisting of: (a) an adaptive boosting technique; (b) a support vector machine technique; (c) a Bayesian network model technique; (d) a neural networks technique; (e) a hidden Markov model technique; and (f) a conditional random fields technique.

17 . An apparatus for processing audio data, the apparatus comprising:
an interface system; and a control system capable of:
receiving, via the interface system, audio data corresponding to a conference recording of a conference involving a plurality of conference participants, the audio data including at least one of: (a) conference participant speech data from multiple endpoints, recorded separately or (b) conference participant speech data from a single endpoint corresponding to multiple conference participants and including information for identifying conference participant speech for each conference participant of the multiple conference participants;
analyzing conversational dynamics of the conference recording to determine conversational dynamics data;
searching the conference recording to determine instances of each of a plurality of segment classifications, each of the segment classifications based, at least in part, on the conversational dynamics data; and
segmenting the conference recording into a plurality of segments, each of the segments corresponding with a time interval and at least one of the segment classifications.

18 . The apparatus of claim 17 , wherein the control system is capable of performing the searching and segmenting processes multiple times at different time scales.

19 . The apparatus of claim 17 , wherein the searching and segmenting processes are based, at least in part, on a hierarchy of segment classifications.

20 .- 21 . (canceled)

22 . A non-transitory medium having software stored thereon, the software including instructions for controlling one or more devices for processing audio data, the software incl

Assignees

• Dolby Laboratories Licensing Corp

Inventors

• Cartwright Richard J
• Li Kai
• Sun Xuejing

Classifications

• G06F40/30

  Semantic analysis · CPC title

• H04M3/56

  Arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities (video conference systems H04N7/15) · CPC title

• G10L15/26

  Speech to text systems (G10L15/08 takes precedence) · CPC title

• H04M2203/303

  Marking · CPC title

• G10L25/48

  specially adapted for particular use · CPC title