Tool for annotating and reviewing audio conversations

Patent metadata

Field	Value
Publication number	US-11765267-B2
Application number	US-202117447039-A
Country	US
Kind code	B2
Filing date	Sep 7, 2021
Priority date	Dec 31, 2020
Publication date	Sep 19, 2023
Grant date	Sep 19, 2023

Abstract

Methods, systems, and computer programs are presented for searching and labeling the content of voice conversations. An Engagement Intelligence Platform (EIP) analyzes conversation transcripts to find states and information for each of the states (e.g., interest rate quoted and value of the interest rate). An annotator User Interface (IU) is provided for performing queries, such as, “Find calls were the agent asked the customer for their name and the customer did not answer;” “Find calls where the customer objected after the interest rate for the loan was quoted, “Find calls where the agent asked for consent for recording the call, but no customer confirmation was received.” The EIP analyzes the conversation and labels (e.g., “tags”) the text where the conversation associated with the label took place, such as, “An interest rate was provided.” The labels are customizable, so each client can define its own labels based on business needs.

First claim

What is claimed is:

1. A method of presenting machine-labeled segments of a transcript, the method comprising:
training, by one or more processors, a machine-learning (ML) model based on training segments of training transcripts, each training segment being associated with one or more training labels;
accessing, by the one or more processors, a conversation transcript that includes text of a conversation held in turns between a first party and a second party;
identifying, by the trained ML model and based on the conversation transcript, a conversation segment including multiple pairs of turns all corresponding to a common topic between the first party and the second party;
labeling, by the trained ML model, the identified conversation segment based on the common topic that corresponds to the multiple pairs of turn included in the identified conversation segment; and
generating a user interface (UI) for presentation on a client device, the UI presenting the conversation transcript with the identified and labeled conversation segment that includes the multiple pairs of turns held between the first party and the second party and corresponding to the common topic.

2. The method of claim 1 , wherein features of the trained ML model comprise at least one of turns identified in the conversation, names, locations of the names in the conversation transcript, values of parameters in the conversation, or a call sentiment for the conversation.

3. The method of claim 1 , wherein a configuration file associated with a user specifies a set of labels that includes the one or more conversation labels associated with the conversation segments.

4. The method of claim 3 , further comprising:
detecting, in the UI, a selection of one or more words of text within the conversation transcript; and
in response to the detecting of the selection, presenting, in the UI, the set of labels for association with the selected one or more words.

5. The method of claim 3 , further comprising:
presenting, in a window of the UI, the set of labels with counters of how many times each label in the set is associated with the conversation transcript.

6. The method of claim 3 , wherein a first label from the set of labels is associated with a parameter, and wherein the trained ML model identifies a value of the parameter within the conversation transcript.

7. The method of claim 3 , further comprising:
accessing the configuration file that specifies the set of labels.

8. The method of claim 1 , wherein a first label among the one or more conversation labels indicates an interest-rate quote being presented in the conversation, and wherein a value of the interest-rate quote is extracted from the conversation transcript and associated with the first label.

9. The method of claim 1 , further comprising:
presenting, in the UI, an option to set a value for an outcome of the conversation, the value being selected from a group consisting of no answer, left message, not interested, and application started.

10. The method of claim 1 , further comprising:
presenting, in the UI, an option to set a value for a sentiment of the conversation, the value being selected from a group consisting of positive, negative, and neutral.

11. A system to present machine-labeled segments of a transcript, the system comprising:
one or more memories; and
one or more processors, communicatively coupled to the one or more memories, configured to perform operations comprising:
training a machine-learning (ML) model based on training segments of training transcripts, each training segment being associated with one or more training labels;
accessing a conversation transcript that includes text of a conversation held in turns between a first party and a second party;
identifying, by the trained ML model and based on the conversation transcript, a conversation segment including multiple pairs of turns all corresponding to a common topic between the first party and the second party;
labeling, by the trained ML model, the identified conversation segment based on the common topic that corresponds to the multiple pairs of turn included in the identified conversation segment; and
generating a user interface (UI) for presentation on a client device, the UI presenting the conversation transcript with the identified and labeled conversation segment that includes the multiple pairs of turns held between the first party and the second party and corresponding to the common topic.

12. The system of claim 11 , wherein features of the trained ML model comprise at least one of turns identified in the conversation, names, locations of the names in the conversation transcript, values of parameters in the conversation, or a call sentiment for the conversation.

13. The system of claim 11 , wherein the operations further comprise:
detecting, in the UI, a selection of one or more words of text within the conversation transcript; and
in response to the detecting of the selection, presenting, in the UI, a set of labels for association with the selected one or more words.

14. The system of claim 11 , wherein the operations further comprise:
presenting, in a window of the UI, a set of labels with counters of how many times each label in the set is associated with the conversation transcript.

15. The system of claim 14 , wherein a first label from the set of labels is associated with a parameter, and wherein the trained ML model identifies a value of the parameter within the conversation transcript.

16. A non-transitory computer-readable medium storing instructions that, when executed by a computing device, cause the computing device to perform operations comprising:
training a machine-learning (ML) model based on training segments of training transcripts, each training segment being associated with one or more training labels;
accessing a conversation transcript that includes text of a conversation held in turns between a first party and a second party;
identifying, by the trained ML model and based on the conversation transcript, a conversation segment including multiple pairs of turns all corresponding to a common topic between the first party and the second party;
labeling, by the trained ML model, the identified conversation segment based on the common topic that corresponds to the multiple pairs of turn included in the identified conversation segment; and
generating a user interface (UI) for presentation on a client device, the UI presenting the conversation transcript with the identified and labeled conversation segment that includes the multiple pairs of turns held between the first party and the second party and corresponding to the common topic.

17. The non-transitory computer-readable medium of claim 16 , wherein features of the trained ML model comprise at least one of turns identified in the conversation, names, locations of the names in the conversation transcript, values of parameters in the conversation, or a call sentiment for the conversation.

18. The non-transitory computer-readable medium of claim 16 , wherein the operations further comprise:
detecting, in the UI, a selection of one or more words of text within the conversation transcript; and
presenting, in the UI, a set of labels for association with the selected one or more words.

19. The non-transitory computer-readable medium of claim 16 , wherein the operations further comprise:
presenting, in a window of the UI, a set of labels with counters of how many times each label in the set is associated with the conversation transcript.

Assignees

• Twilio Inc

Inventors

• Cole Tyler
• Pardo San Llorente Eduardo
• KUMAR Sahil
• Ahn Byeongung
• Lee Elaine
• Acosta Amador Nicolas
• Pérez Diez Pablo

Classifications

• G06F40/279 Primary

  Recognition of textual entities · CPC title

• H04M3/42221 Primary

  Conversation recording systems (at the subscriber's set H04M1/656) · CPC title

• G10L15/26

  Speech to text systems (G10L15/08 takes precedence) · CPC title

• G06F40/169

  Annotation, e.g. comment data or footnotes · CPC title

• G06F40/30

  Semantic analysis · CPC title