Machine-learned language models which generate intermediate textual analysis in service of contextual text generation

Patent metadata

Field	Value
Publication number	US-11574131-B2
Application number	US-202217749844-A
Country	US
Kind code	B2
Filing date	May 20, 2022
Priority date	May 21, 2021
Publication date	Feb 7, 2023
Grant date	Feb 7, 2023

Abstract

The present disclosure is directed to systems and methods that include and/or leverage one or more machine-learned language models that generate intermediate textual analysis (e.g., including usage of structural tools such as APIs) in service of contextual text generation. For example, a computing system can obtain a contextual text string that includes one or more contextual text tokens. The computing system can process the contextual text string with the machine-learned language model to generate one or more intermediate text strings that include one or more intermediate text tokens. The computing system can process the one or more intermediate text strings with the machine-learned language model to generate an output text string comprising one or more output text tokens. The one or more intermediate text strings can include textual analysis of the contextual text string that supports the output text string.

First claim

What is claimed is:

1. A computing system for contextual text generation with improved interpretability, the computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store:
a machine-learned language model that performs textual analysis in service of contextual text generation; and
instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining a contextual text string comprising one or more contextual text tokens;
processing the contextual text string with the machine-learned language model to generate one or more intermediate text strings comprising one or more intermediate text tokens, wherein the one or more intermediate text tokens comprise at least one tool token that invokes use of a structural tool to access additional information not included in the contextual text string; and
processing the one or more intermediate text strings with the machine-learned language model to generate an output text string comprising one or more output text tokens;
wherein the one or more intermediate text strings comprise textual analysis of the contextual text string that supports the output text string.

2. The computing system of claim 1 , wherein the structural tool comprises a database lookup to access additional information from a database.

3. The computing system of claim 1 , wherein the structural tool comprises an application programming interface (API) call to request and receive additional information via the API.

4. The computing system of claim 1 , wherein the structural tool comprises a programming language interpreter that performs a sequence of one or more operations on input text tokens.

5. The computing system of claim 1 , wherein the structural tool comprises a query service that queries results from a search engine, knowledge graph, or digital assistant.

6. The computing system of claim 1 , wherein, when the machine-learned language model generates the tool token, the operations comprise:
pausing the machine-learned language model;
executing the structural tool to access the additional information;
appending the additional information to a current version of the one or more intermediate text strings; and
resuming text generation with the machine-learned language model based on the current version of the one or more intermediate text strings and the appended additional information.

7. The computing system of claim 1 , further comprising processing the output text string with a text to speech system to generate an audio output.

8. The computing system of claim 1 , wherein the machine-learned language model operates on a token-by-token basis and, when generating the one or more intermediate text strings, receives each generated intermediate text token as input in a recursive fashion.

9. The computing system of claim 1 , wherein processing the contextual text string with the machine-learned language model to generate one or more intermediate text strings comprising one or more intermediate text tokens comprises:
for a first iteration:
processing the contextual text string with the machine-learned language model to generate a first intermediate text string comprising one or more intermediate text tokens; and
appending the first intermediate text string to the contextual text string to generate an updated contextual text string; and

for each of one or more additional iterations and until the machine-learned language model outputs a closing token:
processing the updated contextual text string with the machine-learned language model to generate an additional intermediate text string comprising one or more intermediate text tokens; and
appending the additional intermediate text string to the updated contextual text string to generate the updated contextual text string for the next iteration.

10. The computing system of claim 1 , wherein the machine-learned language model has been trained on a plurality of training tuples, each training tuple comprising an example contextual text string, one or more example intermediate text strings, and an example output text string.

11. The computing system of claim 10 , wherein at least the one or more example intermediate text strings have been generated by human labelers.

12. The computing system of claim 1 , wherein the machine-learned language model comprises a question answering model, and wherein the contextual text string comprises a question.

13. The computing system of claim 1 , wherein the machine-learned language model comprises a dialog model, and wherein the contextual text string comprises a dialog history.

14. The computing system of claim 1 , wherein the machine-learned language model comprises:
a recurrent neural network;
a multi-headed self-attention model; or
a sequence-to-sequence model.

15. The computing system of claim 1 , wherein at least a portion of the contextual text string comprises text that was input by a user, and wherein the operations further comprise providing at least the output text string for display to the user.

16. The computing system of claim 1 , wherein the contextual text string comprises an original contextual text string that has been concatenated with a base output generated by a machine-learned language model configured to directly generate the base output from the original contextual text string without generating intermediate text strings.

17. A computer-implemented method for improved contextual text generation, the method comprising:
obtaining a plurality of training tuples, each training tuple comprising an example contextual text string comprising one or more contextual text tokens, one or more example intermediate text strings comprising one or more intermediate text tokens, and an example output text string comprising one or more output text tokens;
for each training tuple:
inputting at least a portion of the training tuple to a language model;
receiving a predicted next token as an output of the language model, the predicted next token generated by the language model by processing the portion of the training tuple;
evaluating a loss function that compares the predicted next token generated by the language model with an actual next token included in the training tuple; and
modifying one or more values of one or more parameters of the language model based on the evaluation of the loss function;

wherein, for at least one of the training tuples, the one or more intermediate text tokens comprise at least one tool token that invokes use of a structural tool to access additional information not included in the example contextual text string, and wherein the additional information is included within the one or more intermediate text tokens.

18. The computer-implemented method of claim 17 , wherein, for each training tuple, said inputting, receiving, evaluating, and modifying are performed for each token included in the one or more example intermediate text strings and the example output text string.

19. A computing system for contextual text generation with improved interpretability, the computing system comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store:
a machine-learned language model that performs textual analysis in service of contextual text generation; and
instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprisin

Assignees

• Google Llc

Inventors

• Shazeer Noam
• De Freitas Adiwardana Daniel

Classifications

• G06F40/35 Primary

  Discourse or dialogue representation · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06F16/90332

  Natural language query formulation or dialogue systems · CPC title

• G06F40/284 Primary

  Lexical analysis, e.g. tokenisation or collocates · CPC title

• G06F40/279

  Recognition of textual entities · CPC title