System and method for scoring performance of chatbots

Patent metadata

Field	Value
Publication number	US-10832002-B2
Application number	US-201815973842-A
Country	US
Kind code	B2
Filing date	May 8, 2018
Priority date	May 8, 2018
Publication date	Nov 10, 2020
Grant date	Nov 10, 2020

Abstract

A method, apparatus, system and a computer program product are provided for scoring performance of a chatbot by capturing specified node performance metrics based on user interactions with one or more nodes in a dialog flow for the chatbot; assigning a score to each of the one or more nodes based on the specified node performance metrics; and assessing a performance measure for at least a portion of the chatbot by combining scores assigned to selected nodes in the dialog flow.

First claim

What is claimed is:

1. A method of scoring performance of a chatbot, the method comprising:
capturing, by an information handling system comprising a processor and a memory, specified node performance metrics based on user interactions with one or more nodes in a dialog flow for the chatbot;
assigning, by the information handling system, a score to each of the one or more nodes based on the specified node performance metrics; and
assessing, by the information handling system, a performance measure for at least a portion of the chatbot by combining scores assigned to selected nodes in the dialog flow.

2. The method of claim 1 , further comprising:
inserting, by the information handling system, detection instruments at selected nodes in the dialog flow to track user communications with said selected nodes node in terms of one or more performance metrics selected from a group consisting of time of node access, access count, content, sentiment, emotion, phase, and outcome of each communication.

3. The method of claim 1 , where the specified node performance metrics are selected from a group consisting of time of node access, access count, content, sentiment, emotion, phase, user type, user state, user information, and outcome of each communication.

4. The method of claim 1 , where capturing specified node performance metrics comprises logging, by the information handling system, conversation chats between one or more users and the chatbot.

5. The method of claim 1 , where assigning the score comprises:
applying, by the information handling system, an algorithm to generate one or more assessments for each node based on the specified node performance metrics; and
applying, by the information handling system, a weighted award or penalty score to each node based on the one or more assessments generated for said node.

6. The method of claim 5 , where each assessment generated for each node is selected from a group consisting of a number and type of node accesses, elapsed time at each node, communication emotion or sentiment or tone, and node outcomes.

7. The method of claim 1 , where assessing the performance measure comprises:
selecting, by the information handling system, a first plurality of nodes defining a dialog flow path; and
adding, by the information handling system, each score assigned to each of the first plurality of nodes to generate a performance score for the dialog flow path.

8. The method of claim 1 , where assessing the performance measure comprises generating a first performance measure by combining scores assigned to selected nodes in the dialog flow based on a user type, user state, or user information.

9. The method of claim 1 , further comprising suggesting, by the information handling system, performance improvements or flags for making changes to one or more nodes in the dialog flow based on the performance measure.

10. The method of claim 1 , further comprising leveraging, by the information handling system, one or more of the scores as learning features for a machine learning process that is applied to improve the dialog flow.

11. An information handling system comprising:
one or more processors;
a memory coupled to at least one of the processors;
a set of instructions stored in the memory and executed by at least one of the processors to score performance of a chatbot, wherein the set of instructions are executable to perform actions of:
capturing, by the system, specified node performance metrics based on user interactions with one or more nodes in a dialog flow for the chatbot;
assigning, by the system, a score to each of the one or more nodes based on the specified node performance metrics; and
assessing, by the system, a performance measure for at least a portion of the chatbot by combining scores assigned to selected nodes in the dialog flow.

12. The information handling system of claim 11 , wherein the set of instructions are executable to insert detection instruments at selected nodes in the dialog flow to track user communications with said selected nodes node in terms of one or more performance metrics selected from a group consisting of time of node access, access count, content, sentiment, emotion, phase, and outcome of each communication.

13. The information handling system of claim 11 , where the specified node performance metrics are selected from a group consisting of time of node access, access count, content, sentiment, emotion, phase, user type, user state, user information, and outcome of each communication.

14. The information handling system of claim 11 , wherein the set of instructions are executable to assign the score to each of the one or more nodes by:
applying, by the system, an algorithm to generate one or more assessments for each node based on the specified node performance metrics; and
applying, by the system, a weighted award or penalty score to each node based on the one or more assessments generated for said node.

15. The information handling system of claim 14 , where each assessment generated for each node is selected from a group consisting of a number and type of node accesses, elapsed time at each node, communication emotion or sentiment or tone, and node outcomes.

16. The information handling system of claim 11 , wherein the set of instructions are executable to assess the performance measure by:
selecting, by the system, a first plurality of nodes defining a dialog flow path; and
adding, by the system, each score assigned to each of the first plurality of nodes to generate a performance score for the dialog flow path.

17. The information handling system of claim 11 , wherein the set of instructions are executable to assess the performance measure by combining scores assigned to selected nodes in the dialog flow to generate a first performance measure based on a user type, user state, or user information.

18. The information handling system of claim 11 , wherein the set of instructions are executable to suggest performance improvements or flags for making changes to one or more nodes in the dialog flow based on the performance measure.

19. The information handling system of claim 11 , wherein the set of instructions are executable to leverage one or more of the scores as learning features for a machine learning process that is applied to improve the dialog flow.

20. A computer program product stored in a computer readable storage medium, comprising computer instructions that, when executed by an information handling system, causes the system to score a performance of a chatbot by:
inserting, by the system, detection instruments at selected nodes in the dialog flow to track user communications with said selected nodes node in terms of one or more performance metrics selected from a group consisting of time of node access, access count, content, sentiment, emotion, phase, and outcome of each communication;
capturing, by the system, specified node performance metrics based on user interactions with one or more nodes in a dialog flow for the chatbot;
assigning, by the system, a score to each of the one or more nodes based on the specified node performance metrics;
assessing, by the system, a performance measure for at least a portion of the chatbot by combining scores assigned to selected nodes in the dialog flow; and
suggesting, by the system, performance improvements or flags for making changes to one or more nodes in the dialog flow based on the performance measure.

Assignees

• IBM

Inventors

• Kozhaya Joseph N
• Anderson Ryan
• Govindjee Anita
• Torres Javier R

Classifications

• H04L51/222

  using geographical location information, e.g. messages transmitted or received in proximity of a certain spot or area · CPC title

• G06F40/35 Primary

  Discourse or dialogue representation · CPC title

• H04L51/02

  using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages · CPC title

• G06F11/3438

  monitoring of user actions (tracking the activity of the user H04L67/535) · CPC title

• G06F11/3466

  Performance evaluation by tracing or monitoring · CPC title