System and method for alerts for missing coverage of chatbot conversation messages

Patent metadata

Field	Value
Publication number	US-11539650-B2
Application number	US-202016930615-A
Country	US
Kind code	B2
Filing date	Jul 16, 2020
Priority date	Jul 16, 2020
Publication date	Dec 27, 2022
Grant date	Dec 27, 2022

Abstract

A method, system, and computer-usable medium are disclosed for identifying areas to improve an interactive conversational system, such as a chatbot. A stream of stream of conversational interactions C (C 1 , C 2 , . . . , C n ) between users and the interactive conversational system is received. An intent clustering model is periodically applied to the stream to form an incremental clustering based on a set of derived intents to form a mapping from a first set of conversational characteristics to a first set of intents and a second set of conversational characteristics to a first set of unclear intents. Information is provided related to the second set of conversation characteristics.

First claim

What is claimed is:

1. A computer-implemented method for identifying areas to improve an interactive conversational system comprising:
receiving a stream of conversational interactions C (C 1 , C 2 , . . . , C n ) between users and the interactive conversational system;
monitoring instances of the conversational interactions C (C 1 , C 2 , . . . , C n ) for intents that do not exist for a set of related messages or questions;
applying periodically an intent clustering model to the stream to form an incremental clustering based on a set of derived intents to form a mapping from a first set of conversational characteristics to a first set of intents and a second set of conversational characteristics to a first set of unclear intents that include the intents that do not exist, wherein changes are made to a set of candidate intents relevant to a set of incoming utterances;
computing semantic similarity between the utterances and centroids of exiting clusters;
adding an utterance to a first cluster that matches a condition;
creating a new cluster if there is no cluster matching the condition, wherein the utterance is the centroid of the new cluster, wherein an intent clustering algorithm is applied to the set of incoming utterances and trained based on previous selections of subject matter experts; and
providing information related to the second set of conversation characteristics.

2. The method of claim 1 , wherein the information includes a frequency and a context for a selected a set of conversations taken from the stream of conversational interactions C (C 1 , C 2 , . . . , C n ).

3. The method of claim 2 , wherein the information further comprises identifying a missing intent and providing a suggestion for adding the missing intent into the system.

4. The method of claim 3 further comprising receiving updates for the missing intent and adding the received updates into the interactive conversational system.

5. The method of claim 1 , wherein multiple interactive conversational system are supported.

6. The method of claim 1 , wherein the receiving the stream is in real-time.

7. The method of claim 1 , wherein the providing the information further includes providing recommendations as to skills that can be applied to the interactive conversational system.

8. A system comprising:
a processor;
a data bus coupled to the processor; and
a computer-usable medium embodying computer program code, the computer-usable medium being coupled to the data bus, the computer program code used for identifying areas to improve an interactive conversational system executable by the processor and configured for:
receiving a stream of conversational interactions C (C 1 , C 2 , . . . , C n ) between users and the interactive conversational system;
monitoring instances of the conversational interactions C (C 1 , C 2 , . . . , C n ) for intents that do not exist for a set of related messages or questions;
applying periodically an intent clustering model to the stream to form an incremental clustering based on a set of derived intents to form a mapping from a first set of conversational characteristics to a first set of intents and a second set of conversational characteristics to a first set of unclear intents that include the intents that do not exist, wherein changes are made to a set of candidate intents relevant to a set of incoming utterances, compute semantic similarity between the utterances;
computing semantic similarity between the utterances and centroids of exiting clusters;
adding an utterance to a first cluster that matches a condition;
creating a new cluster if there is no cluster matching the condition, wherein the utterance is the centroid of the new cluster, wherein an intent clustering algorithm is applied to the set of incoming utterances and trained based on previous selections of subject matter experts; and
providing information related to the second set of conversation characteristics.

9. The system of claim 8 , wherein the information includes a frequency and a context for a selected a set of conversations taken from the stream of conversational interactions C (C 1 , C 2 , . . . , C n ).

10. The system of claim 9 , wherein the information further comprises identifying a missing intent and providing a suggestion for adding the missing intent into the system.

11. The system of claim 10 further comprising receiving updates for the missing intent and adding the received updates into the interactive conversational system.

12. The system of claim 8 , wherein multiple interactive conversational system are supported.

13. The system of claim 8 , wherein the wherein the receiving the stream is in real-time.

14. A non-transitory, computer-readable storage medium embodying computer program code, the computer program code comprising computer executable instructions configured for:
receiving a stream of conversational interactions C (C 1 , C 2 , . . . , C n ) between users and the interactive conversational system;
monitoring instances of the conversational interactions C (C 1 , C 2 , . . . , C n ) for intents that do not exist for a set of related messages or questions;
applying periodically an intent clustering model to the stream to form an incremental clustering based on a set of derived intents to form a mapping from a first set of conversational characteristics to a first set of intents and a second set of conversational characteristics to a first set of unclear intents that include the intents that do not exist, wherein changes are made to a set of candidate intents relevant to a set of incoming utterances;
computing semantic similarity between the utterances and centroids of exiting clusters;
adding an utterance to a first cluster that matches a condition;
creating a new cluster if there is no cluster matching the condition, wherein the utterance is the centroid of the new cluster, wherein an intent clustering algorithm is applied to the set of incoming utterances and trained based on previous selections of subject matter experts; and
providing information related to the second set of conversation characteristics.

15. The non-transitory, computer-readable storage medium of claim 14 , wherein the information includes a frequency and a context for a selected a set of conversations taken from the stream of conversational interactions C (C 1 , C 2 , . . . , C n ).

16. The non-transitory, computer-readable storage medium of claim 15 further comprising receiving updates for the missing intent and adding the received updates into the interactive conversational system.

17. The non-transitory, computer-readable storage medium of claim 14 , wherein the clustering algorithm generates a dendrogram, and identifies a threshold to split the dendrogram into clusters.

18. The non-transitory, computer-readable storage medium of claim 14 , wherein multiple interactive conversational systems are supported.

19. The non-transitory, computer-readable storage medium of claim 14 , wherein the receiving the stream is in real-time.

20. The non-transitory, computer-readable storage medium of claim 14 , wherein the providing the information further includes providing recommendations as to skills that can be applied to the interactive conversational system.

Assignees

• IBM

Inventors

• Wayne Eric Donald
• Croutwater Kyle
• Whitley Michael David
• Verma Vikrant
• Zhang Zhe

Classifications

• H04L51/216

  Handling conversation history, e.g. grouping of messages in sessions or threads · CPC title

• H04L51/02 Primary

  using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages · CPC title

• H04L51/224 Primary

  providing notification on incoming messages, e.g. pushed notifications of received messages · CPC title

• H04L51/04

  Real-time or near real-time messaging, e.g. instant messaging [IM] · CPC title