Generative AI business insight report using LLMs

Patent metadata

Field	Value
Publication number	US-2024420161-A1
Application number	US-202418814816-A
Country	US
Kind code	A1
Filing date	Aug 26, 2024
Priority date	Jun 13, 2023
Publication date	Dec 19, 2024
Grant date	—

Abstract

Systems and methods for Large Language Models (LLMs) to generate an Artificial Intelligence (AI) business insight report using business insight data include obtaining business insight data for an organization where the business insight data is from a plurality of sources including from monitoring of a plurality of users associated with the organization; inputting the business insight data to a first Large Language Model (LLM) to generate an initial output for a business insight report; inputting the initial output to a second LLM for critiquing the initial output against a set of rules to check for predefined flaws and to check for what was done correctly to generate a critique; resolving the initial output and the critique to generate a final output; and providing the final output for the business insight report.

First claim

What is claimed is:

1 . A method comprising steps of:
obtaining business insight data for an organization from monitoring of a plurality of users associated with the organization; inputting the business insight data to a first Large Language Model (LLM) to generate an initial output for a business insight report; inputting the initial output to a second LLM for critiquing the initial output against a set of rules to check for predefined flaws and to check for what was done correctly to generate a critique; resolving the initial output and the critique to generate a final output; and providing the final output for the business insight report.

2 . The method of claim 1 , wherein the business insight report comprises a plurality of sections, the sections including an executive summary, a state of Software-as-a-Service (SaaS) applications associated with the organization, areas of improvement, tabular data summarizing the organizations SaaS portfolio, projected spend, and peer comparisons.

3 . The method of claim 2 , wherein the first LLM is configured to generate the initial output based on a template having the plurality of sections, each section having a predefined structure and one or more rules for generation thereof.

4 . The method of claim 1 , wherein the critique includes a list of both flaws in the initial output and places in the initial output performed correctly.

5 . The method of claim 1 , wherein the set of rules to check for predefined flaws include explicit instructions to check to determine whether the first LLM performed correctly.

6 . The method of claim 1 , wherein the set of rules to check for predefined flaws include grammar, conciseness, and passive voice.

7 . The method of claim 1 , wherein the set of rules to check for what was done correctly include explicit instructions to check to determine whether the first LLM performed correctly.

8 . A non-transitory computer-readable medium comprising instructions that, when executed, cause one or more processors to perform steps of:
obtaining business insight data for an organization from monitoring of a plurality of users associated with the organization; inputting the business insight data to a first Large Language Model (LLM) to generate an initial output for a business insight report; inputting the initial output to a second LLM for critiquing the initial output against a set of rules to check for predefined flaws and to check for what was done correctly to generate a critique; resolving the initial output and the critique to generate a final output; and providing the final output for the business insight report.

9 . The non-transitory computer-readable medium of claim 8 , wherein the business insight report comprises a plurality of sections, the sections including an executive summary, a state of Software-as-a-Service (Saas) applications associated with the organization, areas of improvement, tabular data summarizing the organizations SaaS portfolio, projected spend, and peer comparisons.

10 . The non-transitory computer-readable medium of claim 9 , wherein the first LLM is configured to generate the initial output based on a template having the plurality of sections, each section having a predefined structure and one or more rules for generation thereof.

11 . The non-transitory computer-readable medium of claim 8 , wherein the critique includes a list of both flaws in the initial output and places in the initial output performed correctly.

12 . The non-transitory computer-readable medium of claim 8 , wherein the set of rules to check for predefined flaws include explicit instructions to check to determine whether the first LLM performed correctly.

13 . The non-transitory computer-readable medium of claim 8 , wherein the set of rules to check for predefined flaws include grammar, conciseness, and passive voice.

14 . The non-transitory computer-readable medium of claim 8 , wherein the set of rules to check for what was done correctly include explicit instructions to check to determine whether the first LLM performed correctly.

15 . An apparatus comprising:
one or more processors, and memory storing instructions that, when executed, cause the one or more processors to:
obtain business insight data for an organization from monitoring of a plurality of users associated with the organization;
input the business insight data to a first Large Language Model (LLM) to generate an initial output for a business insight report;
input the initial output to a second LLM for critiquing the initial output against a set of rules to check for predefined flaws and to check for what was done correctly to generate a critique;
resolve the initial output and the critique to generate a final output; and
provide the final output for the business insight report.

16 . The apparatus of claim 15 , wherein the business insight report comprises a plurality of sections, the sections including an executive summary, a state of Software-as-a-Service (SaaS) applications associated with the organization, areas of improvement, tabular data summarizing the organizations SaaS portfolio, projected spend, and peer comparisons.

17 . The apparatus of claim 16 , wherein the first LLM is configured to generate the initial output based on a template having the plurality of sections, each section having a predefined structure and one or more rules for generation thereof.

18 . The apparatus of claim 15 , wherein the critique includes a list of both flaws in the initial output and places in the initial output performed correctly.

19 . The apparatus of claim 15 , wherein the set of rules to check for predefined flaws include explicit instructions to check to determine whether the first LLM performed correctly.

20 . The apparatus of claim 15 , wherein the set of rules to check for predefined flaws include grammar, conciseness, and passive voice.

Assignees

• Zscaler Inc

Inventors

• Shah Raimi
• Krishna Raj
• Gangwar Aman
• Jayan Aditya

Classifications

• G06Q10/06375

  Prediction of business process outcome or impact based on a proposed change · CPC title

• G06Q10/067

  Enterprise or organisation modelling · CPC title

• G06Q30/018 Primary

  Certifying business or products · CPC title

• G06Q10/0635 Primary

  Risk analysis of enterprise or organisation activities · CPC title