Computing confidence scores for large language model-based retrieval augmentation tools

Patent metadata

Field	Value
Publication number	US-2026093735-A1
Application number	US-202418901107-A
Country	US
Kind code	A1
Filing date	Sep 30, 2024
Priority date	Sep 30, 2024
Publication date	Apr 2, 2026
Grant date	—

Abstract

A system for computing confidence scores for large language model (LLM) based retrieval augmentation (RAG) tools includes a host device having a controller. The controller executing programmatic control logic including an algorithm for computing confidence scores for LLM based RAG tools (CLR application). The CLR application receives an input to the LLM from a user and engages an ensemble retriever. The ensemble retriever determines a similarity between the input and predetermined data in the one or more databases. The ensemble retriever generates an output and an output confidence score. The CLR application then causes a human validator to prioritize and review the output in accordance with the output confidence score, where the output is a command to one or more systems of the host device. The system progressively reduces computational resource utilization, progressively increases computational efficiency, and progressively reduces reliance on the human validator over time.

First claim

What is claimed is:

1 . A system for computing confidence scores for large language model (LLM) based retrieval augmentation (RAG) tools, the system comprising:
a host device having a controller, the controller having a processor, a memory, and input/output (I/O) ports, the I/O ports in communication with a human-machine interface (HMI) and one or more databases, the processor executing programmatic control logic stored in the memory, the programmatic control logic including an algorithm for computing confidence scores for LLM based RAG tools (CLR application), the CLR application comprising: a first control logic that receives an input to the LLM from a host device user; a second control logic that engages an ensemble retriever, wherein the ensemble retriever that determines a similarity between the input and predetermined data in the one or more databases stored in the memory; a third control logic that causes the ensemble retriever to generate an output and an output confidence score; and a fourth control logic that causes a human validator to prioritize and review the output in accordance with the output confidence score, wherein the output is a command to one or more systems of the host device, and wherein the system progressively reduces computational resource utilization, progressively increases computational efficiency, and progressively reduces reliance on the human validator over time.

2 . The system of claim 1 wherein the first control logic further comprises:
control logic for receiving the input via the human-machine interface (HMI) of the host device.

3 . The system of claim 1 wherein the second control logic further comprises:
engaging a semantic tool that extracts semantic information from the input; and
wherein the semantic tool converts extracted semantic information from the input into a vector in vector space such that the vector defines a mathematical, graphical, vectorized representation of semantic information within the input.

4 . The system of claim 3 , wherein the semantic tool further comprises:
control logic for accessing a text vector database stored in memory, wherein the text vector database contains a plurality of predefined semantic text vectors defining mathematical, graphical, vectorized representations of predefined semantic inputs that the host device is programmed to accept and respond to; and control logic for comparing predefined semantic text vectors in the text vector database to vectorized extracted semantic information from the input.

5 . The system of claim 4 , wherein the semantic tool further comprises:
control logic for using ranked fusion to calculate a semantic similarity score according to:
a retrieved context C i1 , a vector similarity score, S i , and eliminating context C i when a similarity score of C i is such that the vector similarity score S i is less than a threshold T;
fusing “high quality” contexts defined based on an importance score where importance weight is calculated based on a count of context, C i , in different retrievers, CC i , a rank of a context, C i , in different retrievers, RC i , and an importance score of a context, C i , is given as, IC i =f(CC i , RC i ); and
ranking context based on the importance score IC i according to:

Ret
_a

=

[

(

C_a1
,
S_a1

)

,

(

C_a2
,
S_a2

)

,

…
⁡
(

C
an

,

S
an

)

]

Ret
_b

=

[

(

C_b1
,
S_b1

)

,

(

C_b2
,
S_b2

)

,

…
⁡
(

C
bn

,

S
bn

)

]

Ret
_c

=

[

(

C_c1
,
S_c1

)

,

(

C_c2
,
S_c2

)

,

…
⁡
(

C
cn

,

S
cn

)

]

where, context {C ij }, j≥1 is ranked based on similarity score, i.e.: S i1 >S i2 . . . >S in , wherein “low quality” contexts have vector similarity scores S i less than the threshold T, while “high quality” contexts have importance scores IC i indicating that the input is closely related to or directly implicates critical host device functions.

6 . The system of claim 5 , further comprising:
engaging a syntactic tool that extracts syntactic information from the input; and
wherein the syntactic tool converts extracted syntactic information from the input into an input raw text vector in vector space such that the vector defines a mathematical, graphical, vectorized representation of syntactic information within the input.

7 . The system of claim 6 , further comprising:
control logic for accessing a raw text database stored in memory, wherein the raw text database contains a plurality of predefined raw text vectors defining mathematical, graphical, vectorized representations of predefined raw

Assignees

• Gm Global Tech Operations Llc

Inventors

• Adiththan Arun
• Sethu Ramesh
• Peranandam Prakash M
• Sarwar Azeem
• Carver Howard J
• Dixit Eshan
• Gupte Mihir
• Tayyab Muhammad

Classifications

• G06F16/3347

  using vector based model · CPC title

• G06F40/30

  Semantic analysis · CPC title

• G06F16/3344 Primary

  using natural language analysis · CPC title