Speech recognition using log-linear model

Patent metadata

Field	Value
Publication number	US-10134394-B2
Application number	US-201514708465-A
Country	US
Kind code	B2
Filing date	May 11, 2015
Priority date	Mar 20, 2015
Publication date	Nov 20, 2018
Grant date	Nov 20, 2018

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, relating to generating log-linear models. In some implementations, n-gram parameter values derived from an n-gram language model are obtained. N-gram features for a log-linear language model are determined based on the n-grams corresponding to the obtained n-gram parameter values. A weight for each of the determined n-gram features is determined, where the weight is determined based on (i) an n-gram parameter value that is derived from the n-gram language model and that corresponds to a particular n-gram, and (ii) an n-gram parameter value that is derived from the n-gram language model and that corresponds to an n-gram that is a sub-sequence within the particular n-gram. A log-linear language model having the determined n-gram features is generated, where the determined n-gram features in the log-linear language model have weights that are initialized based on the determined weights.

First claim

What is claimed is:

1. A method performed by one or more computers, the method comprising:
obtaining n-gram parameter values derived from an n-gram language model, the n-gram parameter values including n-gram parameter values for n-grams that include multiple words;
determining n-gram features for a log-linear language model based on the n-grams corresponding to the obtained n-gram parameter values;
determining a weight for each of the determined n-gram features, wherein for at least some of the n-gram features, the weight is determined based on (i) an n-gram parameter value that is derived from the n-gram language model and that corresponds to a particular n-gram of multiple words, and (ii) an n-gram parameter value that is derived from the n-gram language model and that corresponds to an n-gram that is a sub-sequence within the particular n-gram;
generating a log-linear language model having the determined n-gram features, the determined n-gram features in the log-linear language model having weights that are initialized based on the determined weights;
after generating the log-linear language model, training the log-linear language model to adjust the initialized weights;
after training the log-linear language model, using the log-linear language model to determine a transcription for an utterance; and
providing the transcription for the utterance.

2. The method of claim 1 , wherein the n-gram language model is configured to assign, to an n-gram that does not have a corresponding parameter value in the n-gram language model, a score based on a parameter value in the n-gram language model for a sub-sequence of words within the n-gram.

3. The method of claim 1 , wherein, for each of the n-grams that includes multiple words, the corresponding parameter value indicates a conditional probability of an occurrence of a last word in the n-gram given an occurrence of one or more words that precede the last word in the n-gram.

4. The method of claim 1 , wherein generating the log-linear language model comprises generating the log-linear language model to indicate, for one or more n-grams, likelihoods of occurrence that equal to likelihoods of occurrence indicated by the n-gram language model for the one or more n-grams.

5. The method of claim 1 , wherein determining the n-gram features for the log-linear language model comprises determining, for each n-gram parameter value in the n-gram language model, an n-gram feature that represents an occurrence of a particular word in a particular context comprising one or more words.

6. The method of claim 1 , wherein determining the n-gram features for the log-linear language model comprises:
identifying each of the n-grams corresponding to the n-gram parameter values derived from the n-gram language model; and
determining the n-gram features to include a feature corresponding to each identified n-gram.

7. The method of claim 1 , further comprising:
obtaining backoff parameter values derived from the n-gram language model, the backoff parameter values representing n-gram backoffs from one order of n-gram to a lower order of n-gram;
determining backoff features for the log-linear language model that represent the backoffs of the n-grams in the n-gram language model;
determining a weight for each backoff feature in the determined backoff features, wherein each of the weights for the determined backoff features is respectively determined based on a backoff parameter value derived from the n-gram language model; and
wherein generating the log-linear language model comprises generating the log-linear language model to have the backoff features and corresponding weights that are initialized based on the determined weights for the backoff features.

8. The method of claim 7 , wherein the weights for the backoff features in the log-linear language model each represents a probability adjustment equivalent to the adjustment represented by a corresponding backoff parameter value in the n-gram language model.

9. The method of claim 1 , wherein determining the weight for each n-gram feature comprises, for at least some of the weights, incorporating a backoff parameter value from the n-gram language model in the weight determination.

10. The method of claim 1 , wherein the log-linear language model includes non-linguistic features corresponding to one or more aspects of non-linguistic context, the non-linguistic features including features indicative of a user characteristic, a time, a geographic location, an application, or an input field.

11. The method of claim 1 , wherein training the log-linear language model comprises training the log-linear language model using stochastic gradient descent training and using a set of training data that is different from a set of training data used to train the n-gram language model.

12. A system comprising:
one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:
obtaining n-gram parameter values derived from an n-gram language model, the n-gram parameter values including n-gram parameter values for n-grams that include multiple words;
determining n-gram features for a log-linear language model based on the n-grams corresponding to the obtained n-gram parameter values;
determining a weight for each of the determined n-gram features, wherein for at least some of the n-gram features, the weight is determined based on (i) an n-gram parameter value that is derived from the n-gram language model and that corresponds to a particular n-gram of multiple words, and (ii) an n-gram parameter value that is derived from the n-gram language model and that corresponds to an n-gram that is a sub-sequence within the particular n-gram;
generating a log-linear language model having the determined n-gram features, the determined n-gram features in the log-linear language model having weights that are initialized based on the determined weights;
after generating the log-linear language model, training the log-linear language model to adjust the initialized weights;
after training the log-linear language model, using the log-linear language model to determine a transcription for an utterance; and
providing the transcription for the utterance.

13. The system of claim 12 , wherein the operations further comprise:
obtaining backoff parameter values derived from the n-gram language model, the backoff parameter values representing n-gram backoffs from one order of n-gram to a lower order of n-gram;
determining backoff features for the log-linear language model that represent the backoffs of the n-grams in the n-gram language model;
determining a weight for each backoff feature in the determined backoff features, wherein each of the weights for the determined backoff features is respectively determined based on a backoff parameter value derived from the n-gram language model; and
wherein generating the log-linear language model comprises generating the log-linear language model to have the backoff features and corresponding weights that are initialized based on the determined weights for the backoff features.

14. The system of claim 12 , wherein generating the log-linear language model comprises generating the log-linear language model to indicate, for one or more n-grams, likelihoods of occurrence that equal to likelihoods of occurrence indicated by the n-gram language model for the one or more n-grams.

15. The system of claim 12 , wherein determining the n-gram features for the log-linear language model comprises

Assignees

• Google Llc

Inventors

• Caseiro Diamantino Antonio
• Biadsy Fadi

Classifications

• G06F40/216

  using statistical methods · CPC title

• G10L15/197 Primary

  Probabilistic grammars, e.g. word n-grams · CPC title

• G06F17/2715

  Physics · mapped topic