Document processing employing probabilistic topic modeling of documents represented as text words transformed to a continuous space

Patent metadata

Field	Value
Publication number	US-9430563-B2
Application number	US-201213364535-A
Country	US
Kind code	B2
Filing date	Feb 2, 2012
Priority date	Feb 2, 2012
Publication date	Aug 30, 2016
Grant date	Aug 30, 2016

Abstract

A set of word embedding transforms are applied to transform text words of a set of documents into K-dimensional word vectors in order to generate sets or sequences of word vectors representing the documents of the set of documents. A probabilistic topic model is learned using the sets or sequences of word vectors representing the documents of the set of documents. The set of word embedding transforms are applied to transform text words of an input document into K-dimensional word vectors in order to generate a set or sequence of word vectors representing the input document. The learned probabilistic topic model is applied to assign probabilities for topics of the probabilistic topic model to the set or sequence of word vectors representing the input document. A document processing operation such as annotation, classification, or similar document retrieval may be performed using the assigned topic probabilities.

First claim

The invention claimed is:

1. An apparatus comprising:
an electronic data processing device configured to:
perform a modeling method including:
constructing a set of word embedding transforms by operations including generating a term-document matrix whose elements represent occurrence frequencies for text words in documents of a set of documents and include inverse document frequency (IDF) scaling;
applying the set of word embedding transforms to transform text words of a set of documents into K-dimensional word vectors in order to generate sets or sequences of word vectors representing the documents of the set of documents where K is an integer greater than or equal to two; and
learning a probabilistic topic model comprising a mixture model including M mixture components representing M topics using the sets or sequences of word vectors representing the documents of the set of documents wherein the learned probabilistic topic model operates to assign probabilities for the topics of the probabilistic topic model to an input set or sequence of K-dimensional embedded word vectors; and
perform a document processing method including:
applying the set of word embedding transforms to transform text words of an input document into K-dimensional word vectors in order to generate a set or sequence of word vectors representing the input document; and
applying the learned mixture model to the set or sequence of word vectors representing the input document in order to generate one of (1) a vector or histogram of topic probabilities representing the input document or (2) one or more Fisher vectors representing the input document.

2. The apparatus of claim 1 , wherein the document processing method further includes:
annotating the input document with one or more topic labels based on the vector, Fisher vector, or histogram of topic probabilities representing the input document.

3. The apparatus of claim 1 , wherein the document processing method further includes:
identifying one or more documents other than the input document as being similar to the input document based on the vector, Fisher vector, or histogram of topic probabilities representing the input document.

4. The apparatus of claim 1 , wherein the constructing the set of word embedding transforms is by operations further including:
applying a dimensionality reduction algorithm to generate K-dimensional word vectors corresponding to the text words of the term-document matrix.

5. The apparatus of claim 4 , wherein the dimensionality reduction algorithm is selected from the group consisting of singular value decomposition (SVD), random indexing, and co-clustering.

6. The apparatus of claim 4 , wherein:
the set of documents used in the constructing of the set of word embedding transforms comprises two aligned corpora including documents in a first natural language and documents in a second natural language,
the generating of a term-document matrix comprises generating a first term-document matrix for the documents in the first natural language and generating a second term-document matrix for the documents in the second natural language,
the applying of a dimensionality reduction algorithm comprises applying a co-clustering algorithm to both first and second term-document matrices to generate K-dimensional word vectors in the same K-dimensional space corresponding to the text words in both the first and second term-document matrices, and
the learning of a probabilistic topic model comprises learning a probabilistic model using sets or sequences of word vectors representing a set of documents of the first language but not the second language, wherein the learned probabilistic topic model operates to assign probabilities for topics of the probabilistic topic model to an input set or sequence of K-dimensional embedded word vectors generated by applying the set of word embedding transforms to a document in the first language, or the second language, or a combination of the first and second languages.

7. The apparatus of claim 1 , wherein the modeling method further includes extending the set of word embedding transforms by identifying one or more related words of a new word and generating a K-dimensional word vector for the new word by aggregating the K-dimensional word vectors generated for the identified one or more related words by the set of word embedding transforms.

8. The apparatus of claim 1 , wherein the set of word embedding transforms comprise a data structure associating text with corresponding K-dimensional word vector.

9. The apparatus of claim 1 , wherein the probabilistic topic model comprises a Gaussian mixture model (GMM) including K-dimensional Gaussian components, the GMM including M Gaussian components corresponding to the M semantic topics.

10. The apparatus of claim 9 , wherein in the performed document processing method, the applying includes:
applying the learned GMM to generate a vector or histogram of topic probabilities representing the input document.

11. The apparatus of claim 9 , wherein in the performed document processing method, the applying includes:
applying the learned GMM to generate one or more Fisher Vectors representing the input document.

12. A non-transitory storage medium storing instructions executable by an electronic data processing device to perform operations including:
constructing a set of word embedding transforms by applying a dimensionality reduction algorithm to a term-document matrix constructed form a set of training documents and having matrix elements storing word frequencies scaled by a metric indicative of the frequencies of occurrence of the words in the set of documents to generate K-dimensional word vectors corresponding to the text words of the term-document matrix;
performing a modeling method including:
applying the set of word embedding transforms to transform text words of a set of documents into K-dimensional word vectors in order to generate the sets or sequences of word vectors representing the documents where K is an integer greater than or equal to two; and
learning a probabilistic topic model comprising M components representing M topics using the sets or sequences of word vectors representing the documents of the set of documents wherein the learned probabilistic topic model operates to assign probabilities for the topics of the probabilistic topic model to an input set or sequence of K-dimensional embedded word vectors; and

performing a document processing method including:
applying the set of word embedding transforms to transform text words of an input document into corresponding K-dimensional word vector in order to generate a set or sequence of word vectors representing the input document; and
applying the learned probabilistic topic model to the set or sequence of word vectors representing the input document in order to generate a topic model representation of the input document comprising one of (1) a vector or histogram of topic probabilities representing the input document or (2) one or more Fisher vectors representing the input document.

13. The non-transitory storage medium of claim 12 , wherein the topic model comprises a Gaussian mixture model (GMM) including M Gaussian components corresponding to the M topics of the topic model.

14. The non-transitory storage medium of claim 12 , wherein the topic model representation of the input document comprises a vector or histogram of topic probabilities representing the input document.

15. The non-transitory storage medium of claim 12 , wherein the topic model representation of the input document comprises one or more Fisher

Assignees

• Clinchant Stéphane
• Perronnin Florent
• Xerox Corp

Inventors

• Clinchant Stéphane
• Perronnin Florent

Classifications

• G06V30/18152

  Extracting features based on a plurality of salient regional features, e.g. "bag of words" · CPC title

• G06F18/2415

  based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate · CPC title

• G06V30/10

  Character recognition · CPC title

• G06K9/6232

  Physics · mapped topic

• G06K9/4676

  Physics · mapped topic