Supervised training for word embedding

Patent metadata

Field	Value
Publication number	US-2018090128-A1
Application number	US-201615275029-A
Country	US
Kind code	A1
Filing date	Sep 23, 2016
Priority date	Sep 23, 2016
Publication date	Mar 29, 2018
Grant date	—

Abstract

A method and system are provided for training word embedding of domain-specific words. The method includes training, by a processor, a first word embedding, using a general domain corpus, on one or more terms inputted by a user. The method further includes retraining, by the processor, the first word embedding, using a specific domain corpus, for a Neuro-Linguistic Programming task, to create a tuned word embedding. The method also includes training, by the processor, a Neural Network for the Neuro-Linguistic Programming task, using the specific domain corpus. The method additionally includes incorporating, by the processor, the trained Neural Network and tuned word embedding into a Neural Network-based Neuro-Linguistic Programming task. The retraining of the first word embedding and the training of the Neural Network are performed together, and the tuned word embedding is accelerated due to a change in a hyper parameter for domain-specific words.

First claim

What is claimed is:

1 . A method for training word embedding of domain-specific words, comprising:
training, by a processor, a first word embedding, using a general domain corpus, on one or more terms inputted by a user; retraining, by the processor, the first word embedding, using a specific domain corpus, for a Neuro-Linguistic Programming task, to create a tuned word embedding; training, by the processor, a Neural Network for the Neuro-Linguistic Programming task, using the specific domain corpus; and incorporating, by the processor, the trained Neural Network and tuned word embedding into a Neural Network-based Neuro-Linguistic Programming task, wherein the retraining of the first word embedding and the training of the Neural Network are performed together, and wherein the tuned word embedding is accelerated due to a change in a hyper parameter for domain-specific words.

2 . The method of claim 1 , wherein the hyper parameter is selected from the group consisting of a learning rate, a dropout rate, and a weight for a regularization term, and wherein, if the hyper parameter increases, the learning rate increases, and, if the hyper parameter decreases, the regularization term decreases.

3 . The method of claim 1 , wherein the trained Neural Network and the tuned word embedding are used in conjunction with a search engine to produce results in response to the one or more terms inputted by the user.

4 . The method of claim 1 , wherein the retraining of the first word embedding includes incorporating a list of terminologies from the domain specific corpus.

5 . The method of claim 4 , wherein the retraining of the first word embedding includes determining the list of terminologies by:
calculating a term frequency-inverse document frequency for each term in the general domain corpus; calculating a term frequency-inverse document frequency for each term in the specific domain corpus; and selecting terms in which a ratio of the term frequency-inverse document frequency for each word in the general corpus to the term frequency-inverse document frequency for each word in the specific domain corpus is larger than a predefined threshold.

6 . The method of claim 5 , wherein the Neural Network-based Neuro-Linguistic Programming task is a Convolutional Neural Network-based question answering task.

7 . The method of claim 1 , wherein:
the one or more terms inputted by the user represent a query; the trained Neural Network and the tuned word embedding are incorporated into the Neural Network-based Neuro-Linguistic Programming task to obtain a response to the query; and the incorporating of the trained Neural Network and the tuned word embedding into the Neural Network-based Neuro-Linguistic Programming task includes displaying, using a graphical user interface, the response to the user.

8 . A system for training word embedding of domain-specific words, comprising:
a graphical user interface configured to enable a user to input one or more terms; a processor, configured to:
train a first word embedding, using a general domain corpus, on the one or more terms inputted by a user;
retrain the first word embedding, using a specific domain corpus, for a Neuro-Linguistic Programming task, to create a tuned word embedding;
train a Neural Network for the Neuro-Linguistic Programming task, using the specific domain corpus; and
incorporate the trained Neural Network and tuned word embedding into a Neural Network-based Neuro-Linguistic Programming task; and
a memory configured to store the one or more terms and the tuned word embedding, wherein the processor retrains the first word embedding and trains the Neural Network together, and wherein the tuned word embedding is accelerated due to a change in a hyper parameter for domain-specific words.

9 . The system of claim 8 , wherein the hyper parameter is selected from the group consisting of a learning rate, a dropout rate, and a weight for a regularization term, and wherein, if the hyper parameter increases, the learning rate increases, and, if the hyper parameter decreases, the regularization term decreases.

10 . The system of claim 8 , wherein the trained Neural Network and the tuned word embedding are used in conjunction with a search engine to produce results in response to the one or more terms inputted by the user.

11 . The system of claim 8 , wherein the retraining of the first word embedding includes incorporating a list of terminologies from the domain specific corpus.

12 . The system of claim 11 , wherein the processor is further configured to determine the list of terminologies by:
calculating a term frequency-inverse document frequency for each term in the general domain corpus; calculating a term frequency-inverse document frequency for each term in the specific domain corpus; and selecting terms in which a ratio of the term frequency-inverse document frequency for each word in the general corpus to the term frequency-inverse document frequency for each word in the specific domain corpus is larger than a predefined threshold.

13 . The system of claim 12 , wherein the Neural Network-based Neuro-Linguistic Programming task is a Convolutional Neural Network-based question answering task.

14 . The system of claim 8 , wherein:
the one or more terms inputted by the user represent a query; the trained Neural Network and the tuned word embedding are incorporated into the Neural Network-based Neuro-Linguistic Programming task to obtain a response to the query; and the graphical user interface is further configured to display the response to the user.

15 . A non-transitory computer-readable storage medium including a computer-readable program for training word embedding of domain-specific words, wherein the computer-readable program when executed on a computer causes the computer to perform a method comprising the steps of:
training, by a processor, a first word embedding, using a general domain corpus, on one or more terms inputted by a user; retraining, by the processor, the first word embedding, using a specific domain corpus, for a Neuro-Linguistic Programming task, to create a tuned word embedding; training, by the processor, a Neural Network for the Neuro-Linguistic Programming task, using the specific domain corpus; and incorporating, by the processor, the trained Neural Network and tuned word embedding into a Neural Network-based Neuro-Linguistic Programming task, wherein the retraining of the first word embedding and the training of the Neural Network are performed together, and wherein the tuned word embedding is accelerated due to a change in a hyper parameter for domain-specific words.

16 . The computer-readable storage medium of claim 15 , wherein the hyper parameter is selected from the group consisting of a learning rate, a dropout rate, and a weight for a regularization term, and wherein, if the hyper parameter increases, the learning rate increases, and, if the hyper parameter decreases, the regularization term decreases.

17 . The computer-readable storage medium of claim 15 , wherein the retraining of the first word embedding includes incorporating a list of terminologies from the domain specific corpus.

18 . The computer-readable storage medium of claim 17 , wherein the retraining of the first word embedding includes determining the list of terminologies by:
calculating a term frequency-inverse document frequency for each term in the general domain corpus; calculating a term frequency-inverse document frequency for each term in the spec

Assignees

• IBM

Inventors

• Kurata Gakuto
• Suzuki Masayuki
• Tachibana Ryuki

Classifications

• G06F40/20 Primary

  Natural language analysis (semantic analysis of natural language G06F40/30) · CPC title

• G10L15/063 Primary

  Training · CPC title

• G10L15/16

  using artificial neural networks · CPC title

• G10L15/065

  Adaptation · CPC title

• G10L15/18

  using natural language modelling · CPC title