Systems and methods of chained conversational prompt engineering for information retrieval

Patent metadata

Field	Value
Publication number	US-12332925-B2
Application number	US-202418736351-A
Country	US
Kind code	B2
Filing date	Jun 6, 2024
Priority date	Oct 5, 2023
Publication date	Jun 17, 2025
Grant date	Jun 17, 2025

Abstract

A system is provided for processing user queries by using an automated agent and a workflow. The system comprises reusable components that include states, tools, and/or data sources. Based on analysis of a query's content and goals, the system generates a workflow comprising a sequence of states, each state optimized for a subtask and dynamically bound to a selected tool(s) for that specific query. The workflow can provide a structured high-level control, while allowing for flexible selection of the tool(s) for each state of the workflow for that given query. The system produces a result using the structured workflow and selected tools, answering a user's original query.

First claim

The invention claimed is:

1. A method of prompt engineering chained conversational prompts that are used with a large language model (LLM) to retrieve information from a plurality of different documents, the method comprising:
loading a prompt pipeline configuration file that includes a prompt pipeline with a plurality of prompt templates, wherein each one of the plurality of prompt templates includes at least one variable for a disclosure item;
executing the prompt pipeline with a value for the disclosure item;
generating a first prompt, based on one of the plurality of prompt templates, and submitting the first prompt, along with a first document of the plurality of documents, to the LLM to determine instances in which the value for the disclosure item is referenced within the first document;
for each determined instance in which the disclosure item is referenced within the first document:
(a) submitting at least two different prompts, each of which are based on different ones of the plurality of prompt templates, to the LLM and receiving, for each submitted one of the at least two different prompts, a corresponding responsive output;
(b) performing a validation process to validate that at least two of the corresponding responsive outputs received for the at least two different prompts are consistent; and
(c) based on validation of the corresponding responsive outputs received for the at least two different prompts, generating a further validation prompt and submitting the further validation prompt to the LLM to further validate the determined instance, wherein content of the further validation prompt includes at least one data item from a prior prompt response;

based on the prompts submitted to the LLM and/or responses received from the LLM for the submitted prompts in (a)-(c), generating and submitting, to the LLM, a contextual summary prompt and receiving a responsive contextual summary that integrates the conversational context of (a)-(c); and
generating, as part of a graphical user interface, the responsive contextual summary in association with the document for which it is associated.

2. The method of claim 1 , wherein the at least two different prompts is at least three different prompts.

3. The method of claim 1 , further comprising:
generating a graphical user interface that includes:
a first section for displaying content of a selected one of the plurality of different documents,
a second section for display each determined instance in which the disclosure item is referenced within the first document.

4. The method of claim 3 , wherein the graphical user interface further includes, as part of the second section, each generated responsive contextual summary.

5. The method of claim 1 , further comprising:
based on validation of the corresponding responsive outputs received for the at least two different prompts, prompting the LLM to return at least one quantitative metric from the document in which the disclosure item is referenced.

6. The method of claim 5 , wherein the at least one data item is the at least one quantitative metric that has been retrieved from the document.

7. The method of claim 1 , further comprising:
storing a plurality values for different disclosure items, wherein separate instances of the prompt pipeline are executed for each one of the plurality of values for the different disclosure items.

8. The method of claim 7 , further comprising:
receiving a plurality of new documents; and
automatically executing the prompt pipeline for each one of the plurality values for different disclosure items against the plurality of new documents.

9. The method of claim 1 , further comprising:
based on validation of the corresponding responsive outputs received for the at least two different prompts, generating a prompt with at least one constraint to return a response from the document that explicitly contains disclosure of a metric name, a numeric value, a year, and a unit of measurement.

10. The method of claim 9 , wherein the at least one data item is at least one of the metric name, the numeric value, the year, and/or the unit of measurement contained in the response to the prompt with the at least one constraint.

11. A computer system for prompt engineering chained conversational prompts that are used with a large language model (LLM) to retrieve information from a plurality of different documents, the computer system comprising:
a storage system configured to store a prompt pipeline configuration file that includes a prompt pipeline with a plurality of prompt templates, wherein each one of the plurality of prompt templates includes at least one variable for a disclosure item;
at least one hardware processor that is configured to execute instructions that cause the at least one hardware processor to perform operations comprising:
executing the prompt pipeline with a value for the disclosure item;
generating a first prompt, based on one of the plurality of prompt templates, and submitting the first prompt, along with a first document of the plurality of documents, to the LLM to determine instances in which the value for the disclosure item is referenced within the first document;
for each determined instance in which the disclosure item is referenced within the first document:
(a) submitting at least two different prompts, each of which are based on different ones of the plurality of prompt templates, to the LLM and receiving, for each submitted one of the at least two different prompts, a corresponding responsive output;
(b) performing a validation process to validate that at least two of the corresponding responsive outputs received for the at least two different prompts are consistent; and
(c) based on validation of the corresponding responsive outputs received for the at least two different prompts, generating a further validation prompt and submitting the further validation prompt to the LLM to further validate the determined instance, wherein content of the further validation prompt includes at least one data item from a prior prompt response;

based on the prompts submitted to the LLM and/or responses received from the LLM for the submitted prompts in (a)-(c), generating and submitting, to the LLM, a contextual summary prompt and receiving a responsive contextual summary that integrates the conversational context of (a)-(c); and
generating, as part of a graphical user interface, the responsive contextual summary in association with the document for which it is associated.

12. The computer system of claim 11 , wherein the at least two different prompts is at least three different prompts.

13. The computer system of claim 11 , wherein the operations further comprise:
generating a graphical user interface that includes:
a first section for displaying content of a selected one of the plurality of different documents,
a second section for display each determined instance in which the disclosure item is referenced within the first document.

14. A computer system of claim 13 , wherein the graphical user interface further includes, as part of the second section, each generated responsive contextual summary.

15. A computer system of claim 11 , wherein the operations further comprise: based on validation of the corresponding responsive outputs received for the at least two different prompts, prompting the LLM to return at least one quantitative metric from the document in which the disclosure item is referenced.

16. A computer system of claim 15 , wherein the at least one data item is the at least one quantitative metric that has been ret

Assignees

• Nasdaq Inc

Inventors

• Aghajanyan Viktor
• Stiller Michael
• Bornacini Eugenia

Classifications

• G06F16/3322

  using system suggestions (G06F16/3325 takes precedence) · CPC title

• G06V30/413

  Classification of content, e.g. text, photographs or tables · CPC title

• G06V30/412 Primary

  Layout analysis of documents structured with printed lines or input boxes, e.g. business forms or tables · CPC title

• G06F16/3326 Primary

  using relevance feedback from the user, e.g. relevance feedback on documents, documents sets, document terms or passages · CPC title