Database systems and methods for automated conversational responses

Patent metadata

Field	Value
Publication number	US-12475149-B2
Application number	US-202318505988-A
Country	US
Kind code	B2
Filing date	Nov 9, 2023
Priority date	Nov 9, 2023
Publication date	Nov 18, 2025
Grant date	Nov 18, 2025

Abstract

Database systems and methods are provided for managing usage of large language models (LLMs). One method involves determining a numerical representation of a conversational input to a user interface, identifying a semantically similar subset of prior conversational inputs based at least in part on the numerical representation of the conversational input, and determining numerical representations of respective conversational responses generated by a language model responsive to the respective prior conversational input of the semantically similar subset. When the numerical representations of the respective conversational responses satisfy a semantic similarity threshold, the method automatically generates an automated response to the conversational input based at least in part on one or more prior conversational responses and automatically provides the automated response to the user interface responsive to the conversational input.

First claim

What is claimed is:

1 . A method comprising:
determining a numerical representation of a conversational input to a user interface; identifying, based at least in part on the numerical representation of the conversational input, a semantically similar subset of prior conversational inputs previously sent to a large language model-based (LLM-based) service from a plurality of prior conversational inputs having associated conversational responses maintained at a database system, wherein a respective prior conversational input of the semantically similar subset has a respective conversational response generated by the LLM-based service responsive to the respective prior conversational input associated therewith, resulting an associated subset of prior conversational responses provided by the LLM-based service responsive to the semantically similar subset of prior conversational inputs; determining numerical representations of the respective conversational responses of the associated subset of prior conversational responses provided by the LLM-based service; and when the numerical representations of the respective conversational responses satisfy a semantic similarity threshold for homogeneity across the prior conversational responses provided by the LLM-based service:
automatically generating a synthetic LLM response to the conversational input based at least in part on one or more prior conversational responses of the associated subset of prior conversational responses provided by the LLM-based service responsive to the semantically similar subset of prior conversational inputs; and
automatically providing the synthetic LLM response to the user interface responsive to the conversational input.

2 . The method of claim 1 , further comprising removing personal identifying information from the conversational input prior to determining the numerical representation of the conversational input.

3 . The method of claim 2 , further comprising supplementing the synthetic LLM response with the personal identifying information prior to automatically the automated synthetic LLM response to the user interface.

4 . The method of claim 1 , wherein identifying the semantically similar subset of prior conversational inputs comprises identifying a cluster group of prior input prompts previously sent to the LLM-based service having respective numerical representations semantically similar to the numerical representation of the conversational input.

5 . The method of claim 1 , further comprising filtering the semantically similar subset of prior conversational inputs based at least in part on contextual information associated with the conversational input prior to identifying the associated subset of prior conversational responses.

6 . The method of claim 5 , wherein the contextual information includes at least one of an identifier associated with a user providing the conversational input and a database object type associated with the conversational input.

7 . At least one non-transitory machine-readable storage medium that provides instructions that, when executed by at least one processor, are configurable to cause the at least one processor to perform operations comprising:
determining a numerical representation of a conversational input to a user interface; identifying, based at least in part on the numerical representation of the conversational input, a semantically similar subset of prior conversational inputs previously sent to a large language model-based (LLM-based) service from a plurality of prior conversational inputs having associated conversational responses maintained at a database system, wherein a respective prior conversational input of the semantically similar subset has a respective conversational response generated by the LLM-based service responsive to the respective prior conversational input associated therewith, resulting an associated subset of prior conversational responses provided by the LLM-based service responsive to the semantically similar subset of prior conversational inputs; determining numerical representations of the respective conversational responses of the associated subset of prior conversational responses provided by the LLM-based service; and when the numerical representations of the respective conversational responses satisfy a semantic similarity threshold for homogeneity across the prior conversational responses provided by the LLM-based service:
automatically generating a synthetic LLM response to the conversational input based at least in part on one or more prior conversational responses of the associated subset of prior conversational responses provided by the LLM-based service responsive to the semantically similar subset of prior conversational inputs; and
automatically providing the synthetic LLM response to the user interface responsive to the conversational input.

8 . The at least one non-transitory machine-readable storage medium of claim 7 , wherein the instructions are configurable to cause the at least one processor to remove personal identifying information from the conversational input prior to determining the numerical representation of the conversational input.

9 . The at least one non-transitory machine-readable storage medium of claim 8 , wherein the instructions are configurable to cause the at least one processor to supplement the synthetic LLM response with the personal identifying information prior to automatically providing the synthetic LLM response to the user interface.

10 . The at least one non-transitory machine-readable storage medium of claim 7 , wherein:
the semantically similar subset of prior conversational inputs comprises a subset of prior input prompts previously sent to the LLM-based service; and the associated subset of prior conversational responses comprises conversational responses provided by the LLM-based service in response to receiving respective ones of the subset of prior input prompts.

11 . The at least one non-transitory machine-readable storage medium of claim 7 , wherein the instructions are configurable to cause the at least one processor to identify a cluster group of prior input prompts previously sent to the LLM-based service having respective numerical representations semantically similar to the numerical representation of the conversational input.

12 . The at least one non-transitory machine-readable storage medium of claim 7 , wherein the instructions are configurable to cause the at least one processor to filter the semantically similar subset of prior conversational inputs based at least in part on contextual information associated with the conversational input prior to identifying the associated subset of prior conversational responses.

13 . The at least one non-transitory machine-readable storage medium of claim 12 , wherein the contextual information includes at least one of an identifier associated with a user providing the conversational input and a database object type associated with the conversational input.

14 . A computing system comprising:
at least one non-transitory machine-readable storage medium that stores software; and at least one processor, coupled to the at least one non-transitory machine-readable storage medium, to execute the software that implements a large language model (LLM) management service and that is configurable to perform operations comprising:
determining a numerical representation of a conversational input to a user interface;
identifying, based at least in part on the numerical representation of the conversational input, a semantically similar subset of prior conversational inputs pre

Assignees

• Salesforce Inc

Inventors

• Conway John

Classifications

• G06F16/338

  Presentation of query results · CPC title

• G06F40/30 Primary

  Semantic analysis · CPC title

• G06F16/3329 Primary

  Natural language query formulation · CPC title