Systems and methods for mutual learning for topic discovery and word embedding

Patent metadata

Field	Value
Publication number	US-2020293902-A1
Application number	US-201916355622-A
Country	US
Kind code	A1
Filing date	Mar 15, 2019
Priority date	Mar 15, 2019
Publication date	Sep 17, 2020
Grant date	—

Abstract

Described herein are embodiments for systems and methods for mutual machine learning with global topic discovery and local word embedding. Both topic modeling and word embedding map documents onto a low-dimensional space, with the former clustering words into a global topic space and the latter mapping word into a local continuous embedding space. Embodiments of Topic Modeling and Sparse Autoencoder (TMSA) framework unify these two complementary patterns by constructing a mutual learning mechanism between word co-occurrence based topic modeling and autoencoder. In embodiments, word topics generated with topic modeling are passed into auto-encoder to impose topic sparsity for the autoencoder to learn topic-relevant word representations. In return, word embedding learned by autoencoder is sent back to topic modeling to improve the quality of topic generations. Performance evaluation on various datasets demonstrates the effectiveness of the disclosed TMSA framework in discovering topics and embedding words.

First claim

What is claimed is:

1 . A computer-implemented method for mutual learning with topic discovery and word embedding using one or more processors to cause steps to be performed comprising:
receiving input comprising a Dirichlet prior and a document set having at least one document; for each word in the document set:
drawing, from a word embedding matrix, a word embedding for the word, the word embedding matrix is updated using an encoder sparsified with topics to reflect topic distribution of words;
drawing, from a residual matrix, residuals for each word co-occurrence corresponding to the word, each residual presenting nonlinear or noisy interaction between the word and another word in each document;
drawing, from a topic embedding matrix, one or more topic embeddings corresponding to the word;
for each document in the document set:
drawing, from the Dirichlet prior, a mixing topic proportion representing relative proportions among topics for each document;
drawing at least one topic from a topic matrix for a j-th word in each document based on the mixing topic proportion, j is a positive integer number; and
passing the at least one topic drawn for the j-th word into the encoder for updating the word embedding matrix.

2 . The computer-implemented method of claim 1 wherein the word embedding matrix is initialized by pretrained word embeddings.

3 . The computer-implemented method of claim 1 further comprising
configuring a topic loss function for the document set to optimize the steps.

4 . The computer-implemented method of claim 1 wherein the encoder is a sparse autoencoder trained with a word loss function comprising at least a topic sparsity parameter.

5 . The computer-implemented method of claim 4 wherein the word embedding in the word embedding matrix is generated, using the sparse autoencoder, by encoding word co-occurrence of the word with a feedforward propagation.

6 . The computer-implemented method of claim 5 wherein the feedforward propagation comprises parameters of a weight matrix and an embedding bias vector.

7 . The computer-implemented method of claim 4 wherein the word loss function comprises a Kullback-Leibler (KL) divergence related to the topic sparsity parameter.

8 . A computer-implemented method for generating word embedding using one or more processors to cause steps to be performed comprising:
receiving input comprising a Dirichlet prior and a document set having at least one document; for each document:
constructing a word co-occurrence matrix comprising a plurality of word co-occurrences respectively corresponding to a plurality of word-pairs;
encoding, using a sparse autoencoder sparsified with topic information, at least word co-occurrence of an input word in each document to an embedding representation by a feedforward propagation;
decoding, using a decoder, the embedding representation of the input word back to a reconstructed representation;
training the sparse autoencoder by minimizing a word loss function incorporating a topic sparsity parameter.

9 . The computer-implemented method of claim 8 wherein the feedforward propagation comprises parameters of a weight matrix and an embedding bias vector.

10 . The computer-implemented method of claim 9 wherein the word loss function further comprises a Kullback-Leibler (KL) divergence to control sparsity of the weight matrix and embedding bias vector parameters.

11 . The computer-implemented method of claim 8 wherein the word co-occurrence matrix is extracted from a sequence of words in each document of the document set within a text window.

12 . The computer-implemented method of claim 11 wherein the text window is fixed and remains the same across documents.

13 . The computer-implemented method of claim 11 wherein each word sequence has a focus word and its neighboring context words within a text window centered at the focus word.

14 . The computer-implemented method of claim 8 wherein the word loss function comprises a term of Kullback-Leibler (KL) divergence between the topic sparsity parameter and an average activation of topic embeddings.

15 . The computer-implemented method of claim 8 wherein the topic information is drawn from a topic matrix based on a mixing topic proportion, the mixing topic proportion is generated from the Dirichlet prior.

16 . A computer-implemented method for mutual learning with topic discovery and word embedding using one or more processors to cause steps to be performed comprising:
receiving input comprising a Dirichlet prior, a word co-occurrence matrix, and a document set having at least one document; initializing at least a topic matrix, a topic embedding matrix, a residual matrix, a weight matrix for a sparsified autoencoder; generating a mixing topic proportion representing relative proportions among topics based on the Dirichlet prior and the topic embedding matrix; with a word embedding matrix fixed, updating topics in the topic matrix based on at least the mixing topic proportion; encoding, using the sparsified autoencoder sparsified with the updating topics, word co-occurrences in the word co-occurrence matrix to corresponding word embeddings by a feedforward propagation; calculating an overall objection function combined from a topic loss function and a word loss function; and updating the weight matrix for the sparsified autoencoder with backpropagation.

17 . The computer-implemented method of claim 16 further comprising:
updating the word embedding matrix using the sparsified autoencoder with the updated the weight matrix.

18 . The computer-implemented method of claim 16 wherein the word loss function comprises a term of Kullback-Leibler (KL) divergence between a topic sparsity parameter and an average activation of topic embeddings.

19 . The computer-implemented method of claim 16 further comprising:
decoding, using a decoder, the embedding representation of the input word back to a reconstructed representation.

20 . The computer-implemented method of claim 19 wherein the word loss function comprises a term representing an average of reconstruction loss between word embeddings by the feedforward propagation and the reconstructed representation.

Assignees

• Baidu Usa Llc

Inventors

• Li Dingcheng
• Zhang Jingyuan
• Li Ping

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/047

  Probabilistic or stochastic networks · CPC title

• G06N3/0455

  Auto-encoder networks; Encoder-decoder networks · CPC title

• G06N3/0495

  Quantised networks; Sparse networks; Compressed networks · CPC title