Measuring quality of a chatbot response

Patent metadata

Field	Value
Publication number	US-12093319-B2
Application number	US-202117249522-A
Country	US
Kind code	B2
Filing date	Mar 4, 2021
Priority date	Mar 4, 2021
Publication date	Sep 17, 2024
Grant date	Sep 17, 2024

Abstract

A computer-implemented method, computer system, and computer program product for measuring a quality of a chatbot response. The present invention may include receiving one or more classifications, receiving a set of questions in a chatbot to be analyzed, filtering any question from the received set of questions that is not related to an area of expertise of the chatbot, matching at least two questions from the received set of questions to each other, and applying at least one of the one or more classifications to the at least two matched questions. The one or more classifications may be based on a similarity of words and synonyms used in the at least two matched questions. The one or more classifications may be based on a similarity of intents of the at least two matched questions.

First claim

What is claimed is:

1. A computer-implemented method for measuring a quality of a chatbot response, the computer-implemented method comprising:
receiving one or more classifications;
receiving a set of questions in a chatbot to be analyzed;
filtering any question from the received set of questions that is not related to an area of expertise of the chatbot;
matching at least two questions from the received set of questions to each other; and
applying at least one of the one or more classifications to the at least two matched questions.

2. The computer-implemented method of claim 1 , wherein the one or more classifications is based on a similarity of words and synonyms used in the at least two matched questions.

3. The computer-implemented method of claim 1 , wherein the one or more classifications is based on a similarity of intents of the at least two matched questions.

4. The computer-implemented method of claim 1 , wherein the one or more classifications is based on a user's frustration associated with the at least two matched questions.

5. The computer-implemented method of claim 1 , further comprising:
creating metadata associated with the at least two matched questions; and
storing the at least two matched questions with the created metadata.

6. The computer-implemented method of claim 1 , wherein the one or more classifications includes a classification score.

7. The computer-implemented method of claim 1 , further comprising:
calculating an overall accuracy of the chatbot;
comparing the calculated overall accuracy of the chatbot with a predefined chatbot accuracy level; and
generating an accuracy report.

8. A computer system for measuring a quality of a chatbot response, comprising:
one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage media, and program instructions stored on at least one of the one or more computer-readable tangible storage media for execution by at least one of the one or more processors via at least one of the one or more computer-readable memories, wherein the computer system is capable of performing a method comprising:
receiving one or more classifications;
receiving a set of questions in a chatbot to be analyzed;
filtering any question from the received set of questions that is not related to an area of expertise of the chatbot;
matching at least two questions from the received set of questions to each other; and
applying at least one of the one or more classifications to the at least two matched questions.

9. The computer system of claim 8 , wherein the one or more classifications is based on a similarity of words and synonyms used in the at least two matched questions.

10. The computer system of claim 8 , wherein the one or more classifications is based on a similarity of intents of the at least two matched questions.

11. The computer system of claim 8 , wherein the one or more classifications is based on a user's frustration associated with the at least two matched questions.

12. The computer system of claim 8 , further comprising:
creating metadata associated with the at least two matched questions; and
storing the at least two matched questions with the created metadata.

13. The computer system of claim 8 , wherein the one or more classifications includes a classification score.

14. The computer system of claim 8 , further comprising:
calculating an overall accuracy of the chatbot;
comparing the calculated overall accuracy of the chatbot with a predefined chatbot accuracy level; and
generating an accuracy report.

15. A computer program product for measuring a quality of a chatbot response, comprising:
one or more computer-readable tangible storage media and program instructions stored on at least one of the one or more computer-readable tangible storage media, the program instructions executable by a processor to cause the processor to perform a method comprising:
receiving one or more classifications;
receiving a set of questions in a chatbot to be analyzed;
filtering any question from the received set of questions that is not related to an area of expertise of the chatbot;
matching at least two questions from the received set of questions to each other; and
applying at least one of the one or more classifications to the at least two matched questions.

16. The computer program product of claim 15 , wherein the one or more classifications is based on a similarity of words and synonyms used in the at least two matched questions.

17. The computer program product of claim 15 , wherein the one or more classifications is based on a similarity of intents of the at least two matched questions.

18. The computer program product of claim 15 , wherein the one or more classifications is based on a user's frustration associated with the at least two matched questions.

19. The computer program product of claim 15 , further comprising:
creating metadata associated with the at least two matched questions; and
storing the at least two matched questions with the created metadata.

20. The computer program product of claim 15 , wherein the one or more classifications includes a classification score.

Assignees

• IBM

Inventors

• Kalandyk Piotr
• Szczepanik Grzegorz Piotr
• Kompanowski Hubert
• Tkaczyk-Walczak Agnieszka

Classifications

• G06F18/217

  Validation; Performance evaluation; Active pattern learning techniques · CPC title

• G06F16/906

  Clustering; Classification · CPC title

• G06F16/908

  using metadata automatically derived from the content · CPC title

• G06F16/90332 Primary

  Natural language query formulation or dialogue systems · CPC title

• G06F16/3329 Primary

  Natural language query formulation · CPC title