System and method for incorporating user feedback in a large language model-based retrieval augmentation tool

Patent metadata

Field	Value
Publication number	US-2026093606-A1
Application number	US-202418901121-A
Country	US
Kind code	A1
Filing date	Sep 30, 2024
Priority date	Sep 30, 2024
Publication date	Apr 2, 2026
Grant date	—

Abstract

A system for incorporating user feedback in a large language model (LLM) based retrieval augmentation (RAG) tools includes an application for incorporating user feedback (UFA). The UFA receives an input to the LLM from a host device user, generates an LLM output as a response to the input, causes a system user to verify the LLM output and provide user feedback via a human-machine interface (HMI) of the host device, and in response to the user feedback, engages a performance tuning optimizer that modifies the LLM output by: revising domain knowledge, revising system prompts, and performing constrained regeneration of the LLM output. The performance tuning optimizer progressively reduces computational resource utilization, progressively increases computational efficiency, and progressively reduces reliance on human validators and user feedback over time, and wherein the LLM output is a command to one or more systems of the host device.

First claim

What is claimed is:

1 . A system for incorporating user feedback in a large language model (LLM) based retrieval augmentation (RAG) tools, the system comprising:
a host device having a controller, the controller having a processor, a memory, and input/output (I/O) ports, the I/O ports in communication with a human-machine interface (HMI) and one or more databases, the processor executing programmatic control logic stored in the memory, the programmatic control logic including an application for incorporating user feedback (UFA) in LLM based RAG tools, the UFA comprising: a first control logic that receives an input to the LLM from a host device user; a second control logic that generates an LLM output as a response to the input; a third control logic that causes a system user to verify the LLM output and provide user feedback via a human-machine interface (HMI) of the host device; and a fourth control logic that, in response to the user feedback, engages a performance tuning optimizer that modifies the LLM output by: revising domain knowledge, revising system prompts, and performing constrained regeneration of the LLM output, wherein the performance tuning optimizer progressively reduces computational resource utilization, progressively increases computational efficiency, and progressively reduces reliance on human validators and user feedback over time, and wherein the LLM output is a command to one or more systems of the host device.

2 . The system of claim 1 , wherein the first control logic further comprises:
control logic for receiving the input via the human-machine interface (HMI) of the host device.

3 . The system of claim 2 , wherein the second control logic further comprises:
control logic for engaging an ensemble retriever, wherein the ensemble retriever that determines a similarity between the input and predetermined data in the one or more databases stored in the memory; control logic for causing the ensemble retriever to generate the LLM output and an output confidence score; control logic for causing a human validator to prioritize and review the LLM output in accordance with the output confidence score; control logic for assigning the output confidence score to the LLM output, wherein the LLM output commands one or more actuators of the host device to adjust performance of relevant host device systems; control logic that causes the human validator to prioritize and review the LLM output according to the output confidence score and a ranked context; and causing the human validator to selectively update one or more of a text vector database and a raw text database with data obtained from the input, the LLM output, and the output confidence score, and wherein the host device comprises a vehicle and LLM output commands to the one or more actuators of the vehicle alter functionality of the vehicle in accordance with inputs received from the user.

4 . The system of claim 3 , wherein the third control logic further comprises:
control logic that prompts the system user for feedback via a request for confirmation that the LLM output is a sufficiently accurate response to the input or user command, wherein the request for confirmation further comprises: an audiovisual, tactile, verbal, numerical, or alphanumeric request for confirmation, and wherein the sufficiently accurate response is gauged based upon personal preference of the user.

5 . The system of claim 4 , wherein the fourth control logic further comprises:
control logic for engaging a subroutine for revising domain knowledge (SRDK), wherein the SRDK receives the LLM output and the user feedback and determines whether the LLM output has been modified by the user and upon determining that the LLM output has been modified by the user, executes control logic for optimizing a domain knowledge base (DKB).

6 . The system of claim 4 , wherein the control logic for optimizing the DKB further comprises:
control logic that retrieves the confidence score of the LLM output that has been modified by the user, and determines whether the LLM output that has been modified by the user is already present in the DKB, wherein upon determining that the LLM output that has been modified by the user is not already present in the DKB, obtains input from human experts to verify that the LLM output that has been modified by the user correctly added to the DKB; and wherein upon determining that the LLM output that has been modified by the user is already present in the DKB, revises domain knowledge embedded in the DKB, thereby generating an updated DKB containing the LLM output that has been modified by the user.

7 . The system of claim 4 , wherein the fourth control logic further comprises:
control logic for engaging a subroutine for revising system prompts (SRSP), wherein the SRSP receives a prompt revision verification request, the LLM output, and the user feedback and determines whether sufficient evidence exists to implement revisions to system prompts, wherein in order to determine whether sufficient evidence exists, the SRSP compares a host device prompt to the user feedback and determines whether a threshold level of similarity exists between the host device prompt and the user feedback, wherein upon determining that sufficient evidence does exist, executing control logic to rewrite the system prompt, subject to performance testing.

8 . The system of claim 7 , wherein the control logic to rewrite the system prompt further comprises:
control logic that performs regression testing on a rewritten system prompt, wherein the regression testing utilizes test inputs stored in memory of the DKB, the regression testing verifies that new information in rewritten system prompts allow the system to continue functioning without negatively impacting system responses, and wherein upon determining that the rewritten system prompt is functioning properly, executes control logic that updates the system prompt in the DKB, and wherein upon determining that the rewritten system prompt is not functioning properly, continues utilizing user feedback to recursively and continuously rewrite the system prompt, regression test the rewritten system prompt and test for system functionality until the regression testing indicates that the new information allows the system to continue functioning without negatively impacting system responses.

9 . The system of claim 4 , wherein the fourth control logic further comprises:
control logic for executing a subroutine for constrained regeneration (SCR), wherein the SCR utilizes user feedback in response to LLM outputs to revise LLM outputs based on user constraints.

10 . The system of claim 9 , wherein the SCR further comprises:
control logic for reviewing context used by the LLM to generate the LLM output; control logic for determining whether low quality context is present, wherein low quality contexts define low-quality matches between the context used by the LLM to generate the LLM output and a context of the user input, wherein low quality contexts are defined according to user preferences; and control logic that receives user feedback via the HMI indicating that a low-quality context is present, removing the low quality context, and reprioritizing contexts before executing control logic to regenerate an LLM output subject to user feedback constrained context.

11 . A method for incorporating user feedback in a large language model (LLM) based retrieval augmentation (RAG) tools, the method comprising:
executing, by a processor of a controller of a vehicle, programmatic control logic stored within memory of the controller, the controller further having input/output (I/O) ports, the I/O p

Assignees

• Gm Global Tech Operations Llc

Inventors

• Adiththan Arun
• Sethu Ramesh
• Peranandam Prakash M
• Sarwar Azeem
• Carver Howard J
• Ullah Syed Shihab
• Paul Aruna I
• Betz Jacob
• Wojno Robert Michael

Classifications

• G06F11/3409

  for performance assessment · CPC title

• G06N5/022

  Knowledge engineering; Knowledge acquisition · CPC title

• G06F11/3672 Primary

  Test management · CPC title