Embeddings with classes

Patent metadata

Field	Value
Publication number	US-11373042-B2
Application number	US-201916702299-A
Country	US
Kind code	B2
Filing date	Dec 3, 2019
Priority date	Dec 13, 2018
Publication date	Jun 28, 2022
Grant date	Jun 28, 2022

Abstract

Described herein are systems and methods for word embeddings to avoid the need to throw out rare words appearing less than a certain number of times in a corpus. Embodiments of the present disclosure involve group words into clusters/classes for multiple times using different assignments of the vocabulary words to a number of classes. Multiple copies of the training corpus are then generated using the assignments to replace each word with the appropriate class. A word embedding generating model is run on the multiple class corpora to generate multiple class embeddings. An estimate of the gold word embedding matrix is then reconstructed from multiple pairs of assignments, class embeddings, and covariances. Test results show the effectiveness of embodiments of the present disclosure.

First claim

What is claimed is:

1. A computer-implemented method for embedding words of a corpus, the method comprising:
grouping a vocabulary of multiple words in the corpus into a set of blocks of words;
for each block:
selecting a number of classes such that each class appears no less than a threshold frequency in the corpus;
in n iterations, assigning the words in the block into the number of classes to obtain n assignment matrices, n is an integer larger than 1;

given the n assignment matrices, obtaining n input corpuses by replacing words in the corpus with its class identifier;
obtaining n class embeddings by running a word embedding generating model on the n input corpuses; and
reconstructing an estimated word embedding for the vocabulary based on at least the n class embeddings.

2. The computer-implemented method of claim 1 wherein the vocabulary is sorted by frequency.

3. The computer-implemented method of claim 2 wherein:
for each block,
assigning each word having a frequency greater than or equal to the threshold frequency of appearance in the corpus to a class by itself; and
grouping two or more words with the appearance frequency in the corpus less than the threshold frequency together into a class.

4. The computer-implemented method of claim 1 wherein for each iteration, the assignment of words are performed to reduce intersection between words in a class across iterations.

5. The computer-implemented method of claim 4 wherein the step of assigning is performed by hashing.

6. The computer-implemented method of claim 1 wherein each assignment matrix is a sparse matrix with rows having unit length.

7. The computer-implemented method of claim 1 wherein the number of classes is a prime number.

8. The computer-implemented method of claim 1 wherein the reconstructed word embedding is an average of the n class embeddings.

9. The computer-implemented method of claim 1 wherein the reconstructed word embedding is reconstructed from an implementation of averaging from the n class embeddings, n is a number less than the number of words in the vocabulary divided by the number of classes.

10. The computer-implemented method of claim 1 wherein words from different blocks are not assigned to the same class.

11. A computer-implemented method for embedding words of a corpus comprising a number of words, the method comprising:
in n iterations, assigning words in the corpus into a number of classes to obtain n assignment matrices, n is an integer larger than 1, the number of classes is less than the number of words in the corpus;
given the n assignment matrices, obtaining n input corpuses by replacing words in the corpus with its class;
obtaining n class embeddings by running a word embedding generating model on the n input corpuses; and
reconstructing an estimated word embedding for the corpus based at least on the n assignment matrices and the n class embeddings.

12. The computer-implemented method of claim 11 further comprising:
comparing the estimated word embedding with a ground truth embedding of the corpus for evaluation.

13. The computer-implemented method of claim 11 wherein for each iteration, the assignment of words are performed to reduce intersection between words in a class across iterations.

14. The computer-implemented method of claim 11 wherein each assignment matrix is a sparse matrix with rows having unit length.

15. The computer-implemented method of claim 11 wherein the number of classes is a prime number.

16. A computer-implemented method for embedding words of a corpus, the method comprising:
receiving a block comprising a number of words;
determining a number of classes for the block of words, the number of classes is less than the number of words in the block; and
generating a class embedding matrix having the number of class embeddings to represent the block of words, the class embedding matrix uses a storage space less than a memory limit of a processor unit, the class embedding matrix is generated by:
determining a threshold on frequency based on the number of words in the block and the number of classes;
computing a class assignment matrix for the block to assign words in the block into corresponding classes based on an appearing frequency of each word in a corpus and the threshold on frequency;
creating, using the class assignment matrix, a copy of the block, where in the copy of the block, each word in the block is replaced by an appropriate class; and
running a word embedding generating model on the copy of the block to create the class embedding matrix.

17. The computer-implemented method of claim 16 wherein the processor unit is a graphic processor unit (GPU).

18. The computer-implemented method of claim 16 wherein the assignment matrix is a sparse matrix with rows having unit length.

19. The computer-implemented method of claim 16 wherein the threshold on frequency is set as the smallest integral value greater than a division result using the number of words divided by the number of classes.

20. The computer-implemented method of claim 19 wherein computing the class assignment matrix comprising:
assigning each word having an appearing frequency greater than or equal to the threshold on frequency to a class by itself; and
assigning two or more words having an appearing frequency less than the threshold on frequency together into a class.

Assignees

• Baidu Usa Llc

Inventors

• Church Kenneth
• Jun Hee Woo
• Hestness Joel

Classifications

• G06F16/35 Primary

  Clustering; Classification · CPC title

• G06F40/30

  Semantic analysis · CPC title

• G06N5/04

  Inference or reasoning models · CPC title

• G06F16/367

  Ontology · CPC title

• G06F40/284 Primary

  Lexical analysis, e.g. tokenisation or collocates · CPC title