Multilingual embeddings for natural language processing

Patent metadata

Field	Value
Publication number	US-9779085-B2
Application number	US-201514863996-A
Country	US
Kind code	B2
Filing date	Sep 24, 2015
Priority date	May 29, 2015
Publication date	Oct 3, 2017
Grant date	Oct 3, 2017

Abstract

A natural language processing (“NLP”) manager is provided that manages NLP model training. An unlabeled corpus of multilingual documents is provided that span a plurality of target languages. A multilingual embedding is trained on the corpus of multilingual documents as input training data, the multilingual embedding being generalized across the target languages by modifying the input training data and/or transforming multilingual dictionaries into constraints in an underlying optimization problem. An NLP model is trained on training data for a first language of the target languages, using word embeddings of the trained multilingual embedding as features. The trained NLP model is applied for data from a second of the target languages, the first and second languages being different.

First claim

What is claimed is:

1. A non-transitory computer readable medium having instructions stored thereon that, when executed by a processor, cause the processor to manage natural language processing (NLP) model training, the managing comprising:
providing an unlabeled corpus of multilingual documents that span a plurality of target languages;
training a multilingual embedding on the corpus of multilingual documents as input training data, the multilingual embedding being generalized across the target languages by modifying the input training data and/or transforming multilingual dictionaries into constraints in an underlying optimization problem;
training an NLP model on training data for a first language of the target languages, using word embeddings of the trained multilingual embedding as features; and
applying the trained NLP model on data from a second of the target languages, the first and second languages being different.

2. The computer readable medium of claim 1 , the multilingual embedding being generalized across the target languages by modifying the input training data.

3. The computer readable medium of claim 1 , the multilingual embedding being generalized across the target languages by transforming multilingual dictionaries into constraints in an underlying optimization problem.

4. The computer readable medium of claim 2 , wherein modifying the input training data comprises artificial code switching.

5. The computer readable medium of claim 4 , wherein artificial code switching comprises: for each word of the input training data, probabilistically replace the word with a word from a different language from the word's corresponding concept set from a dictionary.

6. The computer readable medium of claim 3 , wherein the training a multilingual embedding comprises making a first update of a first vector, and the transforming multilingual dictionaries into constraints in the underlying optimization problem comprises: after making the first update of the first vector, looking up other words in the multilingual dictionaries based on the first update and updating respective other vectors of the other words such that angles between the first vector and the other vectors are close to each other.

7. The computer readable medium of claim 1 , wherein the multilingual embedding is generalized across the target languages by modifying the input training data and modifying a step of a training algorithm transforming multilingual dictionaries into constraints in an underlying optimization problem.

8. A computer-implemented method for managing natural language processing (NLP) model training, the computer-implemented method comprising:
providing an unlabeled corpus of multilingual documents that span a plurality of target languages;
training a multilingual embedding on the corpus of multilingual documents as input training data, the multilingual embedding being generalized across the target languages by modifying the input training data and/or transforming multilingual dictionaries into constraints in an underlying optimization problem;
training an NLP model on training data for a first language of the target languages, using word embeddings of the trained multilingual embedding as features; and
applying the trained NLP model on data from a second of the target languages, the first and second languages being different.

9. The computer-implemented method of claim 8 , the multilingual embedding being generalized across the target languages by modifying the input training data.

10. The computer-implemented method of claim 8 , the multilingual embedding being generalized across the target languages by transforming multilingual dictionaries into constraints in an underlying optimization problem.

11. The computer-implemented method of claim 9 , wherein modifying the input training data comprises artificial code switching.

12. The computer-implemented method of claim 11 , wherein artificial code switching comprises: for each word of the input training data, probabilistically replace the word with a word from a different language from the word's corresponding concept set from a dictionary.

13. The computer-implemented method of claim 10 , wherein the training a multilingual embedding comprises making a first update of a first vector, and the transforming multilingual dictionaries into constraints in the underlying optimization problem comprises: after making the first update of the first vector, looking up other words in the multilingual dictionaries based on the first update and updating respective other vectors of the other words such that angles between the first vector and the other vectors are close to each other.

14. The computer-implemented method of claim 8 , wherein the multilingual embedding is generalized across the target languages by modifying the input training data and transforming multilingual dictionaries into constraints in an underlying optimization problem.

15. A system comprising:
a memory device configured to store a natural language processing (NLP) management module;
a processing device in communication with the memory device, the processing device configured to execute the NLP management module stored in the memory device to manage NLP model training, the managing comprising:
providing an unlabeled corpus of multilingual documents that span a plurality of target languages;
training a multilingual embedding on the corpus of multilingual documents as input training data, the multilingual embedding being generalized across the target languages by modifying the input training data and/or transforming multilingual dictionaries into constraints in an underlying optimization problem;
training an NLP model on training data for a first language of the target languages, using word embeddings of the trained multilingual embedding as features; and
applying the trained NLP model on data from a second of the target languages, the first and second languages being different.

16. The system of claim 15 , the multilingual embedding being generalized across the target languages by modifying the input training data.

17. The system of claim 15 , the multilingual embedding being generalized across the target languages by transforming multilingual dictionaries into constraints in an underlying optimization problem.

18. The system of claim 16 , wherein modifying the input training data comprises artificial code switching.

19. The system of claim 18 , wherein artificial code switching comprises: for each word of the input training data, probabilistically replace the word with a word from a different language from the word's corresponding concept set from a dictionary.

20. The system of claim 17 , wherein the training a multilingual embedding comprises making a first update of a first vector, and the transforming multilingual dictionaries into constraints in the underlying optimization problem comprises: after making the first update of the first vector, looking up other words in the multilingual dictionaries based on the first update and updating respective other vectors of the other words such that angles between the first vector and the other vectors are close to each other.

Assignees

• Oracle Int Corp

Inventors

• WICK Michael Louis
• KANANI Pallika Haridas
• Pocock Adam Craig

Classifications

• G06F40/242 Primary

  Dictionaries · CPC title

• G06F40/44 Primary

  Statistical methods, e.g. probability models · CPC title

• G06F17/2818 Primary

  Physics · mapped topic

• G06F17/2735

  Physics · mapped topic