Processing natural language queries with API calls and API executions

Patent metadata

Field	Value
Publication number	US-12554752-B2
Application number	US-202318344747-A
Country	US
Kind code	B2
Filing date	Jun 29, 2023
Priority date	Jun 29, 2023
Publication date	Feb 17, 2026
Grant date	Feb 17, 2026

Abstract

Systems and methods are provided for a natural language question answering service to provide answers to natural language questions regarding network-based services or computing domains. The natural language question answering service may receive the natural language question from a customer computing device. An aggregator of the natural language question answering service can retrieve passages from search systems based on the question and generate a prompt. A large language model (LLM) of the natural language question answering service may receive the prompt and provide an answer. The answer may be verified by a verifier of the natural language question answering service. Attribution may be applied to the answers and retrieved passages to produce references, inline citations, and similar questions. A watermarking module of the natural language question answering service may watermark the answer if it is verified.

First claim

What is claimed is:

1 . A system for processing queries, the system configured to:
receive, from a customer computing device and via a user interface (UI) associated with the customer computing device, a question in natural language text, the question regarding operation of a network-based service associated with a customer corresponding to the customer computing device; form, via an aggregator of the system, supplemental search system results based on the received question, wherein the supplemental search system results include one or more documents and portions of the one or more documents; generate a prompt based on the received question and a portion of the supplemental search system results; obtain information corresponding to the customer's use of the network-based service; pass, to a large language model (LLM) subsystem of the system, the prompt and the information corresponding to the customer's use of the network-based service to generate, via the LLM subsystem, one or more answers, wherein the one or more answers include an application programming interface (API) command to the network-based service; send, to the customer computing device, the one or more answers with the API command with a prompt as to whether the API command should be executed; receive, from the customer computing device, a confirmation to run the API command; and execute the API command against the network-based service.

2 . The system as recited in claim 1 , wherein the one or more answers comprise human readable text.

3 . The system as recited in claim 2 , wherein the supplemental search system results are also sent to the customer computing device.

4 . The system as recited in claim 1 , wherein in forming the one or more documents by the aggregator, the aggregator is configured to:
send, to the search system, requests to retrieve the one or more documents; and receive, from the search system, the one or more documents.

5 . A system for processing queries, the system comprising a memory storing computer-executable instructions and a processor configured to execute the computer-executable instructions to:
receive, from a customer computing device, a question regarding operation of a network-based service associated with a customer corresponding to the customer computing device; form, via an aggregator of the system, relevant passages corresponding to search system results based on the received question; obtain information corresponding to the customer's use of the network-based service; pass, to a large language model (LLM) subsystem of the system, a prompt generated based on the question, the relevant passages, and the information corresponding to the customer's use of the network-based service to generate, via the LLM subsystem, one or more answers, wherein the one or more answers include an application programming interface (API) command to the network-based service; execute the API command on the network-based service to completion; and send to the customer computing device, the one or more answers and a notification of completion of the API command.

6 . The system as recited in claim 5 , wherein the relevant passages are also sent to the customer computing device.

7 . The system as recited in claim 5 , wherein the LLM subsystem comprises a Retrieval Augmented Generation (RAG) model.

8 . The system as recited in claim 5 , wherein the relevant passages are retrieved by the aggregator from a plurality of search systems, wherein the plurality of search systems include at least one of:
a search system configured to provide answers related to network-based storage systems; a search system configured to provide answers related to network-based on-demand code execution systems; a search system configured to provide answers related to network-based database systems; or a search system configured to provide answers related to network-based on demand compute systems.

9 . The system as recited in claim 8 , wherein at least one of the plurality of search systems is configured to provide answers related to questions of customers of network-based systems.

10 . The system as recited in claim 8 , wherein at least one of the plurality of search systems is configured to provide answers related to frequently asked questions (FAQ) pages of network-based systems.

11 . The system as recited in claim 8 , wherein the LLM subsystem is configured to:
store information regarding the customers of network-based systems, wherein the information at least describes network-based products the customers are subscribed to and actions taken by the customers regarding the network-based products; and train a machine learning model of the LLM subsystem with the stored customers' information to identify patterns of actions of the customers regarding the network-based products.

12 . The system as recited in claim 5 , wherein prior to sending the one or more answers to the customer computing device, the processor is further configured to execute the computer-executable instructions to: produce, via an attribution component of the system, one or more of (i) references, (ii) inline citations, or (ii) similar questions to the question, based on the one or more answers and the search system results.

13 . A computer-implemented method for processing queries, the method comprising:
receiving, from a customer computing device, a question regarding operation of a network-based service associated with a customer corresponding to the customer computing device; forming, via an aggregator, relevant passages to retrieve from search system results based on the received question; obtaining information corresponding to the customer's use of the network-based service; passing, to a large language model (LLM) subsystem of the system, a prompt generated based on the question, the relevant passages, and the information corresponding to the customer's use of the network-based service to generate, via a large language model (LLM) subsystem, one or more answers, wherein the one or more answers include an application programming interface (API) command to the network-based service; executing the API command on the network-based service to completion; and sending to the customer computing device, the one or more answers and a notification of completion of the API command.

14 . The computer-implemented method of claim 13 , wherein the relevant passages comprise of: documents, text of the documents, pictures of the documents, or video of the documents.

15 . The computer-implemented method of claim 13 , wherein the relevant passages are also sent to the customer computing device.

16 . The computer-implemented method of claim 13 , wherein the one or more answers comprise human readable text.

17 . The computer-implemented method of claim 13 , wherein the LLM subsystem comprises a Retrieval Augmented Generation (RAG) model.

18 . The computer-implemented method of claim 13 further comprising prior to sending the one or more answers to the customer computing device, generating, via a watermarking module, a watermarked version of the one or more answers, wherein the watermarked version of the one or more answers is generated by embedding signals into human readable text associated with the one or more answers such that the embedding signals make the watermarked version of the one or more answers proprietary to the system.

19 . The computer-implemented method of claim 13 , wherein in forming the relevant passages by the aggregator, the aggregator is configur

Assignees

• Amazon Tech Inc

Inventors

• Khosla Sopan
• Kumar Vinayshekhar Bannihatti
• Gangadharaiah Rashmi
• Nadig Deepak Seetharam
• Horsley James W
• Barde Abhijit S

Classifications

• G06F16/3344 Primary

  using natural language analysis · CPC title

• G06F9/547

  Remote procedure calls [RPC]; Web services · CPC title

• G06F16/3329 Primary

  Natural language query formulation · CPC title