Mixture-Of-Expert Approach to Reinforcement Learning-Based Dialogue Management

Patent metadata

Field	Value
Publication number	US-2023376697-A1
Application number	US-202318173495-A
Country	US
Kind code	A1
Filing date	Feb 23, 2023
Priority date	May 19, 2022
Publication date	Nov 23, 2023
Grant date	—

Abstract

Systems and methods for dialogue response prediction can leverage a plurality of machine-learned language models to generate a plurality of candidate outputs, which can be processed by a dialogue management model to determine a predicted dialogue response. The plurality of machine-learned language models can include a plurality of experts trained on different intents, emotions, and/or tasks. The particular candidate output selected may be selected by the dialogue management model based on semantics determined based on a language representation. The language representation can be a representation generated by processing the conversation history of a conversation to determine conversation semantics.

First claim

What is claimed is:

1 . A computing system, the system comprising:
one or more processors; and one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining input data, wherein the input data comprises conversation data descriptive of a conversation;
processing the input data with an encoder model to generate a language representation, wherein the language representation comprises a latent embedding associated with the conversation, wherein the encoder model was trained to map an encoded conversation into a latent distribution;
processing the language representation with one or more machine-learned language models to generate one or more candidate outputs, wherein the one or more candidate outputs comprise one or more candidate utterances;
processing the one or more candidate outputs and the language representation with a dialogue management model to generate dialogue planning data; and
generating a predicted dialogue response based on the dialogue planning data, wherein the predicted dialogue response comprises one or more predicted words associated with the one or more candidate utterances.

2 . The system of claim 1 , wherein processing the language representation with the one or more machine-learned language models to generate the one or more candidate outputs comprises:
processing the language representation with a plurality of expert language models to generate a plurality of candidate outputs, wherein the plurality of candidate outputs are associated with a plurality of candidate utterances; and wherein the predicted dialogue response comprises a selected candidate utterance associated with a selected candidate output of the plurality of candidate outputs, and wherein the selected candidate output is selected by the dialogue management model.

3 . The system of claim 1 , wherein the one or more machine-learned language models comprises an expert language model trained on a particular skill such that the one or more candidate utterances are indicative of the particular skill.

4 . The system of claim 1 , wherein the dialogue management model was trained with reinforcement learning, wherein the reinforcement learning optimizes prediction for full conversations; and
wherein the predicted dialogue response comprises a predicted utterance, and wherein the predicted utterance is responsive to the input data.

5 . The system of claim 1 , wherein the language representation is descriptive of semantics of a conversation history of the conversation, wherein the conversation history comprises a plurality of text strings exchanged.

6 . The system of claim 1 , wherein the language representation is associated with a latent space distribution of a learned latent space.

7 . The system of claim 1 , wherein the language representation is associated with a learned distribution of a latent space, wherein the learned distribution is associated with a particular sentiment.

8 . The system of claim 1 , wherein the dialogue management model is configured to:
determine a conversation intent based on the language representation; and select a particular candidate output based on the particular candidate output being associated with the conversation intent.

9 . A computer-implemented method, the method comprising:
obtaining, by a computing system comprising one or more processors, conversation data, wherein the conversation data is descriptive of a conversation history; processing, by the computing system, the conversation data with a language encoding model to generate a language representation, wherein the language representation is descriptive of semantics associated with the conversation history; processing, by the computing system, the language representation with a plurality of machine-learned language models to generate a plurality of candidate outputs, wherein the plurality of machine-learned language models were trained based on learned sentiment distributions associated with a latent space; and processing, by the computing system, the language representation and the plurality of candidate outputs with a dialogue management model to determine a dialogue response.

10 . The method of claim 9 , wherein a first machine-learned language model of the plurality of machine-learned language models was trained for a first skill, and wherein a second machine-learned language model of the plurality of machine-learned language models was trained for a second skill.

11 . The method of claim 9 , wherein the dialogue management model was trained to select a particular candidate output of the plurality of candidate outputs based at least in part on the language representation, and wherein the plurality of machine-learned language models were trained with ground truth training data.

12 . The method of claim 9 , wherein the language encoding model comprises a stochastic encoder model, wherein the stochastic encoder model comprises an encoder and a latent space distribution, and wherein the stochastic encoder model maps a tokenized conversation history to a latent space to generate a parameterized gaussian distribution.

13 . The method of claim 9 , wherein the plurality of machine-learned language models comprise a plurality of expert models associated with a plurality of emotions.

14 . The method of claim 9 , wherein the plurality of machine-learned language models comprise a plurality of expert models associated with a plurality of tasks.

15 . One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations comprising:
obtaining training data, wherein the training data comprises training conversation data; processing the training conversation data with a language encoding model to generate a language representation; processing the language representation with a plurality of machine-learned language models to generate a plurality of candidate utterances; processing the plurality of candidate utterances with a dialogue management model to determine a predicted dialogue response; providing the predicted dialogue response to a user computing system; receiving additional conversation data from the user computing system, wherein the additional conversation data is descriptive of a conversation occurring after the predicted dialogue response; and adjusting one or more parameters of the dialogue management model based on the additional conversation data.

16 . The one or more non-transitory computer-readable media of claim 15 , wherein the operations further comprise:
determining satisfaction data based at least in part on the additional conversation data, wherein the satisfaction data is descriptive of a level of satisfaction with the predicted dialogue response, wherein the satisfaction data is determined based at least in part on conversation engagement; and adjusting one or more parameters of the dialogue management model based on the satisfaction data.

17 . The one or more non-transitory computer-readable media of claim 15 , wherein the training data comprises one or more ground truth utterances, and wherein the operations further comprise:
evaluating a loss function that evaluates a difference between a particular candidate utterance of the plurality of candidate utterances and the one or more ground truth utterances; and adjusting one o

Assignees

• Google Llc

Inventors

• Chow Yinlam
• Nachum Ofir
• Tulepbergenov Azamat

Classifications

• G06F16/90332

  Natural language query formulation or dialogue systems · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06F16/3334

  Selection or weighting of terms from queries, including natural language queries · CPC title

• G06F16/33295

  in dialogue systems · CPC title

• G06F40/35 Primary

  Discourse or dialogue representation · CPC title