Processing natural language queries for network-based services

Patent metadata

Field	Value
Publication number	US-12505135-B2
Application number	US-202318344698-A
Country	US
Kind code	B2
Filing date	Jun 29, 2023
Priority date	Jun 29, 2023
Publication date	Dec 23, 2025
Grant date	Dec 23, 2025

Abstract

Systems and methods are provided for a natural language question answering service to provide answers to natural language questions regarding network-based services or computing domains. The natural language question answering service may receive the natural language question from a customer computing device. An aggregator of the natural language question answering service can retrieve passages from search systems based on the question and generate a prompt. A large language model (LLM) of the natural language question answering service may receive the prompt and provide an answer. The answer may be verified by a verifier of the natural language question answering service. Attribution may be applied to the answers and retrieved passages to produce references, inline citations, and similar questions. A watermarking module of the natural language question answering service may watermark the answer if it is verified.

First claim

What is claimed is:

1 . A system for processing queries, the system comprising:
one or more computing processors and memories configured to execute computer-executable instructions to:
receive, from a customer computing device and via a user interface (UI) associated with the customer computing device, a question in natural language text;
identify, via an aggregator component of the system and based on the question, supplemental search system information for submission in conjunction with the question to a large language model (LLM) component of the system, wherein the supplemental search system information includes:
an open search based index, wherein the open search based index includes one or more documents from a search system based on submission of the received question; and
a dense index, wherein the dense index identifies portions of the one or more documents as pertaining to the received question;

verify, from the aggregator component of the system, that the supplemental search system information is suitable for submission in conjunction with the question to the LLM component by verification of similarities of the supplemental search system information and the question, wherein verification of similarities of the supplemental search system information and the question comprises determination of whether the supplemental search system information is within a pre-determined minimum similarity by:
determining a similarity score characterizing similarities between the supplemental search system information and the question, and
identifying the similarities of the supplemental search system information and the question as verified if the similarity score is above the pre-defined minimum similarity;

in response to verifying the similarities of the supplemental search system information and the question, generate a prompt based on the received question and including a portion of the verified supplemental search system information;
submit the prompt including the portion of the verified supplemental search system information to the LLM component of the system to generate, via the LLM component of the system, one or more answers based on the generated prompt;
verify, via a verifier component of the system, that the one or more answers meet a threshold characterization of whether the one or more answers were generated in error to the question, based on a training data set associated with the LLM;
in response to verifying that the one or more answers meet the threshold characterization, produce, via an attribution component of the system, one or more of (i) references, (ii) inline citations, or (ii) similar questions to the question, based on the one or more answers and the supplemental search system information;
generate, via a watermarking component of the system, a watermarked version of the one or more answers; and
send to the UI of the customer computing device, the watermarked version of the one or more answers, the supplemental search system information, and references, inline citations, or similar questions produced by the attribution component.

2 . The system as recited in claim 1 , wherein the one or more answers comprise of at least human readable text.

3 . The system as recited in claim 2 , wherein the watermarked version of the one or more answers is generated by embedding signals into the human readable text such that the embedding signals make the watermarked version of the one or more answers proprietary to the system.

4 . The system as recited in claim 1 , wherein in identifying the one or more documents by the aggregator component, the aggregator is configured to:
send, to the search system, requests to retrieve the one or more documents; and receive, from the search system, the one or more documents.

5 . A system for processing queries, the system comprising:
one or more computing processors and memories configured to execute computer-executable instructions to:
receive, from a customer computing device, a question;
identify, via an aggregator component of the system, relevant passages corresponding to search system information for submission in conjunction with the question to a large language model (LLM) component of the system based on the question;
verify, from the aggregator component of the system, that the search system information is suitable for submission in conjunction with the question to the LLM component by verification of similarities of the search system information and the question, wherein verification of similarities of the search system information and the question comprises determination of whether the search system information is within a pre-determined minimum similarity by:
determining a similarity score characterizing similarities between the relevant passages and the question, and
if the similarity score is above the pre-defined minimum similarity, characterizing the similarities of the search system information and the question as verified;

in response to verifying that the search system information is suitable for submission in conjunction with the question to the LLM component including the determination of whether the search system information is within a pre-determined minimum similarity, submit a prompt including a portion of the search system information to the LLM component of the system to generate, via the LLM component of the system, one or more answers based on the question and the verified search system information;
verify, via a verifier component of the system, that the one or more answers were not generated in error; and
in response to verifying that the one or more answers were not generated in error, send to the customer computing device, the verified one or more answers.

6 . The system as recited in claim 5 , wherein the relevant passages are also sent to the customer computing device.

7 . The system as recited in claim 5 , wherein the LLM component comprises of a Retrieval Augmented Generation (RAG) model.

8 . The system as recited in claim 5 , wherein the relevant passages are retrieved by the aggregator from a plurality of search systems, wherein the plurality of search systems are at least one of:
a search system configured to provide answers related to network-based storage systems; a search system configured to provide answers related to network-based on-demand code execution systems; a search system configured to provide answers related to network-based database systems; or a search system configured to provide answers related to network-based on demand compute systems.

9 . The system as recited in claim 8 , wherein at least one of the plurality of search systems is configured to provide answers related to questions of customers of network-based systems.

10 . The system as recited in claim 8 , wherein at least one of the plurality of search systems is configured to provide answers related to frequently asked questions (FAQ) pages of network-based systems.

11 . The system as recited in claim 8 , wherein the LLM component is configured to:
store information regarding the customers of network-based systems, wherein the information at least describes network-based products the customer are subscribed to and actions taken by the customers regarding the network-based products; and train a machine learning model of the LLM with the stored customer information to identify patterns of actions of the customers regarding the network-based products.

12 . The system as recited in claim 5 , wherein the verifier component further comprises of:
a textual overlap module; a textual natural language inference module; a relational natural language inference module

Assignees

• Amazon Tech Inc

Inventors

• Khosla Sopan
• Kumar Vinayshekhar Bannihatti
• Gangadharaiah Rashmi
• Nadig Deepak Seetharam
• Horsley James W
• Barde Abhijit S
• Luckria Brijesh
• Abbas Fawad
• Rangaswamy Bhaskar Hosahalli
• Miller Michael W

Classifications

• G06F16/338

  Presentation of query results · CPC title

• G06F16/3329

  Natural language query formulation · CPC title

• G06N20/00 Primary

  Machine learning · CPC title

• G06F40/30

  Semantic analysis · CPC title

• G06F16/3344 Primary

  using natural language analysis · CPC title