Objective training and evaluation

Patent metadata

Field	Value
Publication number	US-12107995-B2
Application number	US-202318458278-A
Country	US
Kind code	B2
Filing date	Aug 30, 2023
Priority date	Oct 24, 2019
Publication date	Oct 1, 2024
Grant date	Oct 1, 2024

Abstract

A system and method configured to generate a simulated caller dialog including a caller intended issue for a scenario for testing a customer service representative (CSR). A simulated caller dialog is presented to the CSR and a CSR response to the simulated caller dialog is received and includes a CSR interpretation of the caller intended issue to the simulated caller dialog. An understanding determination result based on an intent determination recognition score is generated by an intent determination recognition model is generated in response to a comparison of the CSR interpretation of the caller intended issue matching the caller intended issue in the simulated caller dialog. A CSR score is generated for the scenario based on the understanding determination result. The CSR score is recorded to a database.

First claim

What is claimed is:

1. A computer-implemented method comprising:
receiving an agent's response to at least a portion of a dialog for a scenario, the agent's response including an agent's interpretation of an intended issue in the at least the portion of the dialog;
generating an intent determination recognition score, the intent determination recognition score generated by an intent determination recognition model;
in response to the intent determination recognition score indicating incorrect identification of the intended issue, generating a response suggestion based on the agent's response and providing the response suggestion to the agent, the response suggestion including a behavioral indication to increase empathy in a subsequent response to a subsequent restatement of the at least the portion of the dialog; and
generating an agent score for the scenario based on the intent determination recognition score and an empathy recognition score generated by an empathy recognition model, the empathy recognition model coupled to receive the agent's response, the empathy recognition score generated at least in part based on an evaluation of word content in the agent's response to the dialog.

2. The computer-implemented method of claim 1 , wherein the dialog is simulated dialog for the scenario, and wherein the simulated dialog includes a known intended issue.

3. The computer-implemented method of claim 1 , further comprising:
generating accolade feedback to the agent in response to the agent correctly identifying the intended issue.

4. The computer-implemented method of claim 1 , further comprising:
restating the at least the portion of the dialog in response to the agent incorrectly identifying the intended issue in the response.

5. The computer-implemented method of claim 1 , wherein the intent determination recognition model generating the intent determination recognition score comprises:
comparing words in the agent's response with words in the at least the portion of the dialog; and
generating the intent determination recognition score in response to an outcome of the comparison of the words in the agent's response with the words in the at least the portion of the dialog.

6. The computer-implemented method of claim 5 , further comprising:
generating a facial emotional recognition score using a facial emotional recognition model, the facial emotional recognition score in response to the facial emotional recognition model determining a level of emotional recognition in the agent response to the at least the portion of the dialog; and
augmenting the agent score based on the facial emotional recognition score.

7. The computer-implemented method of claim 5 , further comprising:
generating an empathy keyword usage recognition score using an empathy keyword recognition model, the empathy keyword usage recognition score in response to the empathy keyword recognition model determining a level of empathy keyword usage in the agent's response to the at least the portion of the dialog; and
augmenting the agent score based on the empathy keyword usage recognition score.

8. The computer-implemented method of claim 5 , further comprising:
generating an expected keyword usage recognition score using an expected keyword recognition model, the expected keyword usage recognition score using the expected keyword recognition model determining a level of expected keyword usage in the agent's response to the at least the portion of the dialog; and
augmenting the agent score based on the expected keyword usage recognition score.

9. The computer-implemented method of claim 1 , wherein the generating of the agent score further comprises:
generating a leaderboard providing the agent score of the agent with respect to previous scores of other agents.

10. The computer-implemented method of claim 1 , wherein the intended issue relates to one of a member, patient, or service provider.

11. A system, comprising:
one or more processors; and
a memory storing instructions that, when executed, cause the one or more processors to:
receive an agent's response to at least a portion of a dialog for a scenario, the agent's response including an agent's interpretation of an intended issue in the at least the portion of the dialog;
generate an intent determination recognition score, the intent determination recognition score generated by an intent determination recognition model;
in response to the intent determination recognition score indicating incorrect identification of the intended issue, generate a response suggestion based on the agent's response and providing the response suggestion to the agent, the response suggestion including a behavioral indication to increase empathy in a subsequent response to a subsequent restatement of the at least the portion of the dialog; and
generate an agent score for the scenario based on the intent determination recognition score and an empathy recognition score generated by an empathy recognition model, the empathy recognition model coupled to receive the agent's response, the empathy recognition score generated at least in part based on an evaluation of word content in the agent's response to the dialog.

12. The system of claim 11 , wherein the dialog is simulated dialog for the scenario, and wherein the simulated dialog includes a known intended issue.

13. The system of claim 11 , wherein the instructions further comprise instructions to:
generating accolade feedback to the agent in response to the agent correctly identifying the intended issue.

14. The system of claim 11 , wherein the instructions further comprise instructions to:
restating the at least the portion of the dialog in response to the agent incorrectly identifying the intended issue in the response.

15. The system of claim 11 , wherein the intent determination recognition model generating the intent determination recognition score comprises:
comparing words in the agent's response with words in the at least the portion of the dialog; and
generating the intent determination recognition score in response to an outcome of the comparison of the words in the agent's response with the words in the at least the portion of the dialog.

16. The system of claim 15 , wherein the instructions further comprise instructions to:
generating a facial emotional recognition score using a facial emotional recognition model, the facial emotional recognition score in response to the facial emotional recognition model determining a level of emotional recognition in the agent response to the at least the portion of the dialog; and
augmenting the agent score based on the facial emotional recognition score.

17. The system of claim 15 , wherein the instructions further comprise instructions to:
generating an empathy keyword usage recognition score using an empathy keyword recognition model, the empathy keyword usage recognition score in response to the empathy keyword recognition model determining a level of empathy keyword usage in the agent's response to the at least the portion of the dialog; and
augmenting the agent score based on the empathy keyword usage recognition score.

18. The system of claim 15 , wherein the instructions further comprise instructions to:
generating an expected keyword usage recognition score using an expected keyword recognition model, the expected keyword usage recognition score using the expected keyword recognition model determining a level of expected keyword usage in the agent's response to the at least the portion of the dialog; and
augment

Assignees

• Cvs Pharmacy Inc

Inventors

• Caron Roger A
• Daniher Patrick J
• Hays Christopher K
• Livingston Joseph
• Prawl Cadesha M

Classifications

• G06V40/176

  Dynamic expression · CPC title

• H04M2203/401

  Performance feedback · CPC title

• G10L2015/088

  Word spotting · CPC title

• H04M2203/403

  Agent or workforce training · CPC title

• G10L25/63

  for estimating an emotional state · CPC title