Dialog management for large language model- based (LLM-based) dialogs

Patent metadata

Field	Value
Publication number	US-12406147-B2
Application number	US-202318123141-A
Country	US
Kind code	B2
Filing date	Mar 17, 2023
Priority date	Mar 13, 2023
Publication date	Sep 2, 2025
Grant date	Sep 2, 2025

Abstract

Implementations relate to dialog management of a large language model (LLM) utilized in generating natural language (NL) output during an ongoing dialog. Processor(s) of a system can: receive NL based input as part of the ongoing dialog, generate NL based output utilizing the LLM, and cause the NL based output to be rendered. Further, the processor(s) can receive subsequent NL based input as part of the ongoing dialog. In some implementations, the processor(s) can determine whether to modify a corresponding dialog context in generating subsequent NL based output, and modify the corresponding dialog context accordingly. For example, the processor(s) can restrict the corresponding dialog context, or supplant the corresponding dialog context with a corresponding curated dialog context. In additional or alternative implementations, the processor(s) can modify a corresponding NL based output threshold utilized in generating the subsequent NL based response to ensure the resulting NL based output is desirable.

First claim

What is claimed is:

1. A method implemented by one or more processors, the method comprising:
receiving natural language (NL) based input associated with a client device, the NL based input being received during a given turn of an ongoing dialog;
generating, based on processing the NL based input using a large language model (LLM), NL based output that is responsive to the NL based input;
causing the NL based output that is responsive to the NL based input to be rendered at the client device;
receiving subsequent NL based input associated with the client device, the subsequent NL based input being received during a given subsequent turn of the ongoing dialog;
determining, based on the NL based output and/or the subsequent NL based input, whether to modify a corresponding NL based output threshold for generating subsequent NL based output that is responsive to the subsequent NL based input received during the given subsequent turn of the ongoing dialog; and
in response to determining to modify the corresponding NL based output threshold for generating the subsequent NL based output that is responsive to the subsequent NL based input received during the given subsequent turn of the ongoing dialog:
modifying the corresponding NL based output threshold to generate a corresponding modified NL based output threshold for generating the subsequent NL based output that is responsive to the subsequent NL based input received during the given subsequent turn of the ongoing dialog;
generating, based on processing the subsequent NL based input using the LLM and based on the corresponding modified NL based output threshold for generating the subsequent NL based output that is responsive to the subsequent NL based input received during the given subsequent turn of the ongoing dialog, the subsequent NL based output that is responsive to the subsequent NL based input; and
causing the subsequent NL based output that is responsive to the subsequent NL based input to be rendered at the client device.

2. The method of claim 1 , wherein modifying the corresponding NL based output threshold to generate the corresponding modified NL based output threshold for generating the subsequent NL based output that is responsive to the subsequent NL based input received during the given subsequent turn of the ongoing dialog comprises:
increasing an assurance score threshold for generating the subsequent NL based output that is responsive to the subsequent NL based input received during the given subsequent turn of the ongoing dialog.

3. The method of claim 2 , further comprising:
storing the corresponding modified NL based output threshold in a dialog history of a user of the client device; and
utilizing the corresponding modified NL based output threshold in conducting future dialogs with the user of the client device.

4. The method of claim 1 , further comprising:
determining, based on at least the NL based output and/or the subsequent NL based input, whether to modify a corresponding dialog context for the given subsequent turn of the ongoing dialog; and
in response to determining to modify the corresponding dialog context for the given subsequent turn of the ongoing dialog:
modifying the corresponding dialog context for the given subsequent turn of the ongoing dialog to generate a corresponding modified dialog context for the given subsequent turn of the ongoing dialog;
generating, based on processing the subsequent NL based input and the corresponding modified dialog context for the given subsequent turn of the ongoing dialog using the LLM and based on the corresponding modified NL based output threshold for generating the subsequent NL based output that is responsive to the subsequent NL based input received during the given subsequent turn of the ongoing dialog, the subsequent NL based output that is responsive to the subsequent NL based input; and
causing the subsequent NL based output that is responsive to the subsequent NL based input to be rendered at the client device.

5. The method of claim 4 , wherein modifying the corresponding dialog context for the given subsequent turn of the ongoing dialog to generate the corresponding modified dialog context for the given subsequent turn of the ongoing dialog comprises:
restricting the corresponding dialog context to a given prior turn of the ongoing dialog that occurred prior to the given turn of an ongoing dialog to generate the corresponding modified dialog context for the given subsequent turn of the ongoing dialog.

6. The method of claim 5 , wherein restricting the corresponding dialog context to the given prior turn of the ongoing dialog that occurred prior to the given turn of an ongoing dialog to generate the corresponding modified dialog context for the given subsequent turn of the ongoing dialog comprises:
omitting at least the given turn of the ongoing dialog from the corresponding modified dialog context for the given subsequent turn of the ongoing dialog.

7. The method of claim 4 , wherein modifying the corresponding dialog context for the given subsequent turn of the ongoing dialog to generate the corresponding modified dialog context for the given subsequent turn of the ongoing dialog comprises:
supplanting the corresponding dialog context with a corresponding curated dialog context to generate the corresponding modified dialog context for the given subsequent turn of the ongoing dialog.

8. The method of claim 7 , wherein supplanting the corresponding dialog context with the corresponding curated dialog context to generate the corresponding modified dialog context for the given subsequent turn of the ongoing dialog comprises:
selecting, based on corresponding output content captured in the NL based output and/or corresponding subsequent input content captured in the subsequent NL based input, and from among a plurality of curated dialog contexts, the corresponding curated dialog context.

9. A system comprising:
at least one processor; and
memory storing instructions that, when executed, cause the at least one processor to be operable to:
receive natural language (NL) based input associated with a client device, the NL based input being received during a given turn of an ongoing dialog;
generate, based on processing the NL based input using a large language model (LLM), NL based output that is responsive to the NL based input;
cause the NL based output that is responsive to the NL based input to be rendered at the client device;
receive subsequent NL based input associated with the client device, the subsequent NL based input being received during a given subsequent turn of the ongoing dialog;
determine, based on the NL based output and/or the subsequent NL based input, whether to modify a corresponding NL based output threshold for generating subsequent NL based output that is responsive to the subsequent NL based input received during the given subsequent turn of the ongoing dialog; and
in response to determining to modify the corresponding NL based output threshold for generating the subsequent NL based output that is responsive to the subsequent NL based input received during the given subsequent turn of the ongoing dialog:
modify the corresponding NL based output threshold to generate a corresponding modified NL based output threshold for generating the subsequent NL based output that is responsive to the subsequent NL based input received during the given subsequent turn of the ongoing dialog;
generate, based on processing the subsequent NL based input using the LLM and based on the corresponding modified NL based output threshold for generating the subsequent NL based output that is responsive to the subsequent NL based input received during the given subsequent

Assignees

• Google Llc

Inventors

• Baeuml Martin
• Bailey Alexander
• Bragagnolo Jonas
• D'Halluin Florent
• Strohman Trevor

Classifications

• G06N20/00 Primary

  Machine learning · CPC title

• G06F40/56

  Natural language generation · CPC title

• G06F40/35 Primary

  Discourse or dialogue representation · CPC title