Insights into performance of a bot system

Patent metadata

Field	Value
Publication number	US-12242539-B2
Application number	US-202318137829-A
Country	US
Kind code	B2
Filing date	Apr 21, 2023
Priority date	Apr 26, 2019
Publication date	Mar 4, 2025
Grant date	Mar 4, 2025

Abstract

The present disclosure relates generally to techniques for analyzing and improving a bot system, and more particularly to an analytic system integrated with a bot system for monitoring, analyzing, visualizing, diagnosing, and improving the performance of the bot system. For example, an analytic system is integrated with a bot system for monitoring, analyzing, visualizing, and improving the performance of the bot system. The analytic system monitors events occurred in conversations between end users and the bot system, aggregates and analyzes the collected events, and provides information regarding the conversations graphically on a graphic user interface as insights reports at different generalization levels. The insights reports offer developer-oriented analytics to pinpoint issues with skills so a user can address them before they cause problems. The insights let a user track conversation trends over time, identify execution paths, determine the accuracy of their intent resolutions, and access entire conversation transcripts.

First claim

What is claimed is:

1. A method comprising:
accessing a plurality of events generated by a bot system, the bot system comprising a plurality of bots, each bot of the plurality of bots implemented using one or more trained machine-learning models;
generating a plurality of reports based on the plurality of events;
receiving an indication of a selection of a report of the plurality of reports, the indication of the selection received in response to a user providing an input on a graphical user interface, wherein the report provides a conversation completion performance indicator for providing insight into a percentage of complete conversations performed by the plurality of bots, a conversation completion performance indicator for providing insight into a number of complete conversations performed by each respective bot of the plurality of bots, and a conversation incompletion performance indicator for providing insight into a number of incomplete conversations performed by each respective bot of the plurality of bots; and
in response to receiving the indication of the selection of the report, displaying the report and a plurality of user-selectable items for interacting with the report, wherein a user-selectable item of the plurality of user-selectable items comprises an element for selecting a portion of a conversation between a user of the bot system and a bot of the plurality of bots.

2. The method of claim 1 , wherein at least one event of the plurality of events is generated in response to the bot system determining one or more intents for a conversation between a user and one or more bots of the bot system.

3. The method of claim 1 , wherein a report of the plurality of reports provides insight into conversations performed of the plurality of bots over a period of time.

4. The method of claim 1 , wherein the report includes a graphic for evaluating performance of one or more bots of the plurality of bots.

5. The method of claim 1 , wherein the report includes information for evaluating types of errors in performance of one or more bots of the plurality of bots.

6. The method of claim 1 , wherein the conversation is included in a dialog between one or more bots of the bot system and the user.

7. The method of claim 1 , wherein the element for selecting the portion of the conversation comprises a plurality of sub-elements for filtering one or more conversational elements from the conversation.

8. A system comprising:
a processing system; and
one or more computer readable storage media storing instructions which, when executed by the processing system, cause the system to perform a method comprising:
accessing a plurality of events generated by a bot system, the bot system comprising a plurality of bots, each bot of the plurality of bots implemented using one or more trained machine-learning models;
generating a plurality of reports based on the plurality of events;
receiving an indication of a selection of a report of the plurality of reports, the indication of the selection received in response to a user providing an input on a graphical user interface, wherein the report provides a conversation completion performance indicator for providing insight into a percentage of complete conversations performed by the plurality of bots, a conversation completion performance indicator for providing insight into a number of complete conversations performed by each respective bot of the plurality of bots, and a conversation incompletion performance indicator for providing insight into a number of incomplete conversations performed by each respective bot of the plurality of bots; and
in response to receiving the indication of the selection of the report, displaying the report and a plurality of user-selectable items for interacting with the report, wherein a user-selectable item of the plurality of user-selectable items comprises an element for selecting a portion of a conversation between a user of the bot system and a bot of the plurality of bots.

9. The system of claim 8 , wherein at least one event of the plurality of events is generated in response to the bot system determining one or more intents for a conversation between a user and one or more bots of the bot system.

10. The system of claim 8 , wherein a report of the plurality of reports provides insight into conversations performed of the plurality of bots over a period of time.

11. The system of claim 8 , wherein the report includes a graphic for evaluating performance of one or more bots of the plurality of bots.

12. The system of claim 8 , wherein the report includes information for evaluating types of errors in performance of one or more bots of the plurality of bots.

13. The system of claim 8 , wherein the conversation is included in a dialog between one or more bots of the bot system and the user.

14. The system of claim 8 , wherein the element for selecting the portion of the conversation comprises a plurality of sub-elements for filtering one or more conversational elements from the conversation.

15. One or more non-transitory computer-readable media storing computer-readable instructions that, when executed by a processing system, cause a system to perform a method comprising:
accessing a plurality of events generated by a bot system, the bot system comprising a plurality of bots, each bot of the plurality of bots implemented using one or more trained machine-learning models;
generating a plurality of reports based on the plurality of events;
receiving an indication of a selection of a report of the plurality of reports, the indication of the selection received in response to a user providing an input on a graphical user interface, wherein the report provides a conversation completion performance indicator for providing insight into a percentage of complete conversations performed by the plurality of bots, a conversation completion performance indicator for providing insight into a number of complete conversations performed by each respective bot of the plurality of bots, and a conversation incompletion performance indicator for providing insight into a number of incomplete conversations performed by each respective bot of the plurality of bots; and
in response to receiving the indication of the selection of the report, displaying the report and a plurality of user-selectable items for interacting with the report, wherein a user-selectable item of the plurality of user-selectable items comprises an element for selecting a portion of a conversation between a user of the bot system and a bot of the plurality of bots.

16. The one or more non-transitory computer-readable media of claim 15 , wherein at least one event of the plurality of events is generated in response to the bot system determining one or more intents for a conversation between a user and one or more bots of the bot system.

17. The one or more non-transitory computer-readable media of claim 15 , wherein the report includes a graphic for evaluating performance of one or more bots of the plurality of bots.

18. The one or more non-transitory computer-readable media of claim 15 , wherein the report includes information for evaluating types of errors in performance of one or more bots of the plurality of bots.

19. The one or more non-transitory computer-readable media of claim 15 , wherein the conversation is included in a dialog between one or more bots of the bot system and the user.

20. The one or more non-transitory computer-readable media of claim 15 , wherein the element for selecting the por

Assignees

• Oracle Int Corp

Inventors

• Subramaniam Srikant
• Shenbagam Sundar
• Bhatt Neerja
• Kumar Ganesh

Classifications

• G10L2015/0631

  Creating reference templates; Clustering · CPC title

• G10L15/063

  Training · CPC title

• G06N5/041

  Abduction · CPC title

• G06N5/045

  Explanation of inference; Explainable artificial intelligence [XAI]; Interpretable artificial intelligence · CPC title

• G06N20/00

  Machine learning · CPC title