Systems and methods for mutual learning for topic discovery and word embedding

Patent metadata

Field	Value
Publication number	US-11568266-B2
Application number	US-201916355622-A
Country	US
Kind code	B2
Filing date	Mar 15, 2019
Priority date	Mar 15, 2019
Publication date	Jan 31, 2023
Grant date	Jan 31, 2023

Abstract

Described herein are embodiments for systems and methods for mutual machine learning with global topic discovery and local word embedding. Both topic modeling and word embedding map documents onto a low-dimensional space, with the former clustering words into a global topic space and the latter mapping word into a local continuous embedding space. Embodiments of Topic Modeling and Sparse Autoencoder (TMSA) framework unify these two complementary patterns by constructing a mutual learning mechanism between word co-occurrence based topic modeling and autoencoder. In embodiments, word topics generated with topic modeling are passed into auto-encoder to impose topic sparsity for the autoencoder to learn topic-relevant word representations. In return, word embedding learned by autoencoder is sent back to topic modeling to improve the quality of topic generations. Performance evaluation on various datasets demonstrates the effectiveness of the disclosed TMSA framework in discovering topics and embedding words.

First claim

What is claimed is:

1. A computer-implemented method for mutual learning with topic discovery and word embedding using one or more processors to cause steps to be performed comprising:
receiving input comprising a Dirichlet prior and a document set having at least one document;
for each word in a set of words from the document set:
drawing, from a word embedding matrix, a word embedding for the word, the word embedding matrix is updated using an autoencoder sparsified with topics to reflect topic distribution of words, the autoencoder is trained using a word cost function that comprises a sparsity term comprising a first Kullback-Leibler (KL) divergence involving a sparsity parameter and a topic guidance term comprising a second KL divergence involving a topic sparsity parameter, the topic guidance term uses topic distributions of words in the document set to update word embeddings in the word embedding matrix;
drawing, from a residual matrix, residuals for each word co-occurrence corresponding to the word, each residual presenting nonlinear or noisy interaction between the word and another word in each document; and
drawing, from a topic embedding matrix, one or more topic embeddings corresponding to the word;

for each document in the document set:
drawing, from the Dirichlet prior, a mixing topic proportion representing relative proportions among topics for each document;
drawing at least one topic from a topic matrix for a j-th word in each document based on the mixing topic proportion, j is a positive integer number; and
drawing a word from a vocabulary according to a probability of the word given the at least one topic drawn from the topic matrix;

updating one or more topic representations by optimizing a likelihood function for topic, the likelihood function for topic is a function of the document set, the topic matrix, the residual matrix, the word embedding matrix, and the topic embedding matrix; and
outputting the updated word embedding matrix and the one or more updated topic representations.

2. The computer-implemented method of claim 1 wherein the word embedding matrix is initialized by pretrained word embeddings.

3. The computer-implemented method of claim 1 further comprising:
using the updated word embedding matrix and the one or more updated topic representations for document classification.

4. The computer-implemented method of claim 1 wherein the autoencoder is a sparse autoencoder.

5. The computer-implemented method of claim 4 wherein the word embedding in the word embedding matrix is generated, using the sparse autoencoder, by encoding word co-occurrence probabilities of the word with a feedforward propagation, the word co-occurrence probabilities are obtained by counting a number of times each context word occurs around its focus word divided by a frequency of the focus word.

6. The computer-implemented method of claim 5 wherein the word embedding matrix is initialized by pre-trained word embeddings.

7. The computer-implemented method of claim 4 wherein the second KL divergence is between the topic sparsity parameter for a topic and an average activation of the embeddings for the topic.

8. A computer-implemented method for generating word embedding using one or more processors to cause steps to be performed comprising:
receiving input comprising a Dirichlet prior and a document set having at least one document; and
for each document:
constructing a word co-occurrence matrix comprising a plurality of word co-occurrences probabilities respectively corresponding to a plurality of word-pairs;
encoding, using a sparse autoencoder sparsified with topic information, at least word co-occurrence of each input word in each document to a word embedding representation by a feedforward propagation;
decoding, using the sparse autoencoder, the embedding representation of the input word back to a reconstructed representation; and
training the sparse autoencoder to update the word embeddings by minimizing a word loss function that comprises a sparsity term comprising a first Kullback-Leibler (KL) divergence involving a sparsity parameter and a topic guidance term comprising a second KL divergence involving a topic sparsity parameter, the topic guidance term uses topic distribution of words to encapsulate topic information such that the updated word embeddings reflect topic distribution of words.

9. The computer-implemented method of claim 8 wherein the updated word embeddings are used for document classification.

10. The computer-implemented method of claim 9 wherein the second KL divergence is between the topic sparsity parameter for a topic and an average activation of the embeddings for the topic.

11. The computer-implemented method of claim 8 wherein the word co-occurrence matrix is extracted from a sequence of words in each document of the document set within a text window.

12. The computer-implemented method of claim 11 wherein the text window is fixed and remains the same across documents.

13. The computer-implemented method of claim 11 wherein each word sequence has a focus word and its neighboring context words within a text window centered at the focus word.

14. The computer-implemented method of claim 8 wherein the topic information is updated using the updated word embeddings.

15. The computer-implemented method of claim 8 wherein the topic information is drawn from a topic matrix based on a mixing topic proportion, the mixing topic proportion is generated from the Dirichlet prior.

16. A computer-implemented method for mutual learning with topic discovery and word embedding using one or more processors to cause steps to be performed comprising:
receiving input comprising a Dirichlet prior, a word co-occurrence matrix, and a document set having at least one document;
initializing at least a topic matrix, a topic embedding matrix, a residual matrix, a weight matrix for a sparsified autoencoder;
generating a mixing topic proportion representing relative proportions among topics based on the Dirichlet prior and the topic embedding matrix;
with the word embedding matrix fixed, updating topics in the topic matrix based on at least the mixing topic proportion;
encoding, using the sparsified autoencoder sparsified with the updated topics, word co-occurrences in the word co-occurrence matrix to corresponding word embeddings by a feedforward propagation;
training the sparsified autoencoder using an overall objective function combined from a topic loss function and a word loss function, the word loss function comprises a sparsity term comprising a first Kullback-Leibler (KL) divergence involving a sparsity parameter and a topic guidance term comprising a second KL divergence involving a topic sparsity parameter, the topic guidance term uses topic distributions of words in the document set to update word embeddings in the word embedding matrix;
updating the weight matrix for the sparsified autoencoder with backpropagation; and
updating the word embedding matrix using the sparsified autoencoder with the updated the weight matrix.

17. The computer-implemented method of claim 16 wherein the topic loss function is a likelihood function of the document set, the topic matrix, the residual matrix, the word embedding matrix, and the topic embedding matrix.

18. The computer-implemented method of claim 16 wherein the second KL divergence is between the topic sparsity parameter and an average activation of topic embeddings.

19. The computer-implemented method of claim 16 further comprising:
decodin

Assignees

• Baidu Usa Llc

Inventors

• Li Dingcheng
• Zhang Jingyuan
• Li Ping

Classifications

• G06F17/18

  for evaluating statistical data {, e.g. average values, frequency distributions, probability functions, regression analysis (forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04)} · CPC title

• G06N20/00

  Machine learning · CPC title

• G06N3/088 Primary

  Non-supervised learning, e.g. competitive learning · CPC title

• G06N3/084

  Backpropagation, e.g. using gradient descent · CPC title

• G06F17/16

  Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} · CPC title