Prompt refinement to improve accuracy of outputs from machine learning models

Patent metadata

Field	Value
Publication number	US-12524508-B2
Application number	US-202519188116-A
Country	US
Kind code	B2
Filing date	Apr 24, 2025
Priority date	Apr 11, 2024
Publication date	Jan 13, 2026
Grant date	Jan 13, 2026

Abstract

Systems and methods for restructuring prompts in order to improve accuracy of outputs from models are disclosed herein. The system receives a user prompt indicating a request for data. The system generates a first and second output using a model, the first output generated based on the user prompt and the second output generated based on pseudocode. The system compares the first and second outputs to determine a match accuracy between the two outputs. If the two outputs sufficiently match, the system approves the user prompt. If the two outputs do not sufficiently match, the system initiates a prompt restructuring process, whereby the user prompt is restructured using pseudocode to improve the accuracy of the first output. The process is repeated iteratively until the restructured user prompt generates a first output that sufficiently matches the second output generated based on the pseudocode.

First claim

We claim:

1 . One or more non-transitory, computer-readable storage media comprising instructions recorded thereon, wherein the instructions, when executed by at least one data processor of a system, cause the system to:
receive a user prompt indicating a request for a report summarizing data over a time period; generate a first output by inputting, into a generative model, the user prompt to cause the generative model to generate the first output based on the user prompt, the first output comprising a first plurality of text-based analytics, a first plurality of visual analytics, and a first plurality of queries; generate a second output by causing the system to:
retrieve, from a database associated with the data requested in the user prompt, a rule-based pseudocode;
generate a pseudocode prompt based on the rule-based pseudocode and the data requested in the user prompt; and
input, into the generative model, the pseudocode prompt to cause the generative model to generate the second output based on the pseudocode prompt, the second output comprising a second plurality of text-based analytics, a second plurality of visual analytics, and a second plurality of queries;
perform a comparison between (i) the first plurality of text-based analytics and the second plurality of text-based analytics, (ii) the first plurality of visual analytics and the second plurality of visual analytics, and (iii) the first plurality of queries and the second plurality of queries, to determine a match accuracy between the first output and the second output; determine whether the match accuracy satisfies an accuracy threshold; based on the match accuracy failing to satisfy the accuracy threshold, generate a restructured prompt based on the user prompt and the rule-based pseudocode; and approve the restructured prompt for use in conjunction with the generative model.

2 . The one or more non-transitory, computer-readable storage media of claim 1 , wherein, to approve the restructured prompt, the instructions further cause the system to:
generate an updated first output by inputting, into the generative model, the restructured prompt to cause the generative model to generate the updated first output based on the restructured prompt, the updated first output comprising an updated first plurality of text-based analytics, an updated first plurality of visual analytics, and an updated first plurality of queries; perform an updated comparison of the updated first output and the second output to determine an updated match accuracy; determine whether the updated match accuracy satisfies the accuracy threshold; and based on the updated match accuracy satisfying the accuracy threshold, approve the restructured prompt.

3 . The one or more non-transitory, computer-readable storage media of claim 2 , wherein performing the updated comparison comprises causing the system to compare (i) the updated first plurality of text-based analytics with the second plurality of text-based analytics, (ii) the updated first plurality of visual analytics with the second plurality of visual analytics, and (iii) the updated first plurality of queries with the second plurality of queries.

4 . The one or more non-transitory, computer-readable storage media of claim 1 , wherein performing the comparison comprises causing the system to determine a measure of similarity between (i) the first plurality of text-based analytics and the second plurality of text-based analytics, (ii) the first plurality of visual analytics and the second plurality of visual analytics, and (iii) the first plurality of queries and the second plurality of queries.

5 . The one or more non-transitory, computer-readable storage media of claim 1 , wherein the instructions further cause the system to:
determine a ranking of categories of the first output and the second output according to a measure of importance of each category, the categories comprising text-based analytics, visual analytics, and queries,
wherein the comparison is performed according to the ranking.

6 . The one or more non-transitory, computer-readable storage media of claim 5 , wherein, to perform the comparison according to the ranking, the instructions further cause the system to:
compare a higher-ranked category between the first output and the second output, wherein the categories of the first output and the second output comprise at least a higher-ranked category and a lower-ranked category; based on determining that the higher-ranked category fails to match between the first output and the second output, generate the restructured prompt; and based on determining that the higher-ranked category matches between the first output and the second output, approve the user prompt.

7 . A method comprising:
receiving a user prompt indicating a request for data over a time period; generating a first output by inputting, into a generative model, the user prompt to cause the generative model to generate the first output based on the user prompt, the first output comprising a first plurality of text-based analytics, a first plurality of visual analytics, and a first plurality of queries; generating a second output by:
retrieving, from a database associated with the data requested in the user prompt, a structured pseudocode;
generating a pseudocode prompt based on the structured pseudocode and the data requested in the user prompt; and
inputting, into the generative model, the pseudocode prompt to cause the generative model to generate the second output based on the pseudocode prompt, the second output comprising a second plurality of text-based analytics, a second plurality of visual analytics, and a second plurality of queries;
performing a comparison of the first output and the second output; and based on the comparison indicating that the first output does not match the second output, generating a restructured prompt based on the user prompt and the structured pseudocode.

8 . The method of claim 7 , wherein performing the comparison further comprises:
determining, based on the comparison, a match accuracy between the first output and the second output; and determining whether the match accuracy satisfies an accuracy threshold,
wherein the restructured prompt is generated based on the comparison indicating that the match accuracy does not satisfy the accuracy threshold.

9 . The method of claim 8 , further comprising:
generating an updated first output by inputting, into the generative model, the restructured prompt to cause the generative model to generate the updated first output based on the restructured prompt, the updated first output comprising an updated first plurality of text-based analytics, an updated first plurality of visual analytics, and an updated first plurality of queries; performing an updated comparison of the updated first output and the second output to determine an updated match accuracy; determining whether the updated match accuracy satisfies the accuracy threshold; and based on the updated match accuracy satisfying the accuracy threshold, approving the restructured prompt.

10 . The method of claim 9 , wherein performing the updated comparison comprises comparing (i) the updated first plurality of text-based analytics with the second plurality of text-based analytics, (ii) the updated first plurality of visual analytics with the second plurality of visual analytics, and (iii) the updated first plurality of queries with the second plurality of queries.

11 . The method of claim 7 , wherein performing the comparison comprises comparing (i) the first plurality of text-based analytics with the second plurality of text-based analytics, (ii)

Assignees

• Citibank Na

Inventors

• Jeyashekar Nigil Satish
• Engelbrecht Jason
• Wang Zheyu
• Jin Haolin
• Jain Payal
• Maonah Tariq Husayn
• Saternus Mariusz
• Lewandowski Daniel
• Rath Biraj Krushna
• Murray Stuart
• Davies Philip
• Deb Sourabh

Classifications

• G06F40/20

  Natural language analysis (semantic analysis of natural language G06F40/30) · CPC title

• G06F21/6218

  to a system of files or objects, e.g. local or distributed file system or database · CPC title

• G06F21/31 Primary

  User authentication · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G06N3/0455 Primary

  Auto-encoder networks; Encoder-decoder networks · CPC title