Call center agent performance scoring and sentiment analytics

Patent metadata

Field	Value
Publication number	US-10839335-B2
Application number	US-201816218623-A
Country	US
Kind code	B2
Filing date	Dec 13, 2018
Priority date	Dec 13, 2018
Publication date	Nov 17, 2020
Grant date	Nov 17, 2020

Abstract

A system and method for generating an agent behavioral analytics including extracting text-based features, sentiment-based features, and prosody-based features from tagged calls; training a machine learning behavioral model, based on the text-based features, the sentiment-based features, and the prosody-based features extracted from the tagged calls and an at least one score associated with an at least one behavioral metric of the tagged calls, to produce a trained machine learning behavioral model; extracting text-based features, sentiment-based features, and prosody-based features from an incoming call; and using the trained machine learning behavioral model to produce an at least one behavioral label for the agent in the incoming call for the at least one behavioral metric, based on the text-based features of the incoming call, the sentiment-based features of the incoming call and the prosody-based features of the incoming call.

First claim

What is claimed is:

1. A method for generating a behavioral metric for evaluating an agent in a call center, the method comprising:
during a training phase:
obtaining, by a processor, a plurality of audio recordings of tagged calls between customers and agents, and a plurality of scores associated with each tagged call of the plurality of tagged calls, each score grading a tagged call with respect to an attribute of a plurality of attributes;
for each tagged call of the plurality of tagged calls:
for each attribute of an at least one attribute, obtaining, by the processor, a score of the tagged call associated with the attribute; and

transcribing, by the processor, the call using automatic speech recognition and calculating features of the tagged call based on the transcribed text of the tagged call and on the audio recordings of the tagged calls; and
training, by the processor, a neural network implemented machine learning model to produce a behavioral metric for each attribute of the at least one attribute of a future call, based on the features of the tagged calls and the at least one score associated with each of the tagged calls;

during a runtime phase, for an incoming call:
obtaining, by the processor, an audio recording of the call;
transcribing, by the processor, the incoming call and calculating features for the incoming call based on the transcribed text of the incoming call and on the audio recording of the call, wherein the features of a call of the tagged calls and the incoming call comprise a reformulate dialog-act feature, and
calculating the reformulate dialog-act feature by the processor by:
calculating an embedding vector in a word-embedding space for each word in the transcribed text of the call;
dividing the text to utterances of the agent and utterances of the customer;
summing the individual embedding vectors for each word in an utterance to obtain a vector representation of each utterance of a customer and the agent in the word-embedding space;
producing a set of similarity scores by measuring a similarity of the vector representations of each pair of consecutive utterances of the agent and the customer; and

calculating the reformulate dialog act metric based on the similarity scores; and
using, by the processor, the trained neural network implemented machine learning model to produce a grade for the agent in the incoming call for each of the at least one attribute, based on the features of the incoming call.

2. The method of claim 1 , wherein the features of a call of the tagged calls and the incoming call comprise prosody-based features.

3. The method of claim 1 , wherein measuring the similarity of the vector representations is performed using a cosine similarity metric

similarity

=

cos
⁡

(
θ
)

=

A
·
B


A


⁢


B


=

∑

i
=
1

n

⁢

⁢

A
i

⁢

B
i

∑

i
=
1

n

⁢

⁢

A
i
2

⁢

∑

i
=
1

n

⁢

⁢

B
i
2

where A and B are the two utterance vectors in a customer-agent utterance pair.

4. The method of claim 1 , wherein calculating the reformulate dialog-act metric comprises one of:
calculating an average of the similarity measures; and
taking a maximum.

5. The method of claim 1 , wherein the features further comprise at least one of:
number of types,
number of tokens,
type-token ratio calculated by dividing a number of different words in the transcribed text of the call by a total number of words in the transcribed text of the call,
average word length,
average sentence length,
number of words that could not be decoded during transcription,
average confidence level,
number of discourse markers,
silence ratio calculated by dividing total silent time in the call by a total call duration,
an agent activity ratio, calculated by dividing a total agent activity time by a total customer activity time in the call,
a customer activity ratio, calculated by dividing a total customer activity time by a total customer activity time in the call,
a plurality of silence percent per chunk features, wherein a silence percent per chunk feature is calculated by splitting the call to a plurality of chunks and calculating a feature for each chunk by dividing a total silent time in the chunk by a total chunk time,
call duration,
number of repeating unigrams words within a window,
number of repeating unigrams part-of-speech tags within a window,
number of filler words;
number of repeating bigrams words within a window,
number of repeating bigrams part-of-speech tags within a window, and
number of filler words.

6. The method of claim 1 , wherein the features further comprise at least one of:
a repeat-phrase dialog-act feature calculated by dividing a number of matching unigram and bigrams in the pair by a total number of unigrams and bigrams in the pair; and
a back-channel dial

Assignees

• Nice Ltd

Inventors

• Weisman Hila
• Gonen Raanan
• Daoud Rana

Classifications

• G06N20/20

  Ensemble learning · CPC title

• G10L25/48

  specially adapted for particular use · CPC title

• G10L15/26

  Speech to text systems (G10L15/08 takes precedence) · CPC title

• G06Q10/06398 Primary

  Performance of employee with respect to a job function · CPC title

• G10L13/10

  Prosody rules derived from text; Stress or intonation · CPC title