Semi-supervised learning of word embeddings

Patent metadata

Field	Value
Publication number	US-9659560-B2
Application number	US-201514870204-A
Country	US
Kind code	B2
Filing date	Sep 30, 2015
Priority date	May 8, 2015
Publication date	May 23, 2017
Grant date	May 23, 2017

Abstract

Software that trains an artificial neural network for generating vector representations for natural language text, by performing the following steps: (i) receiving, by one or more processors, a set of natural language text; (ii) generating, by one or more processors, a set of first metadata for the set of natural language text, where the first metadata is generated using supervised learning method(s); (iii) generating, by one or more processors, a set of second metadata for the set of natural language text, where the second metadata is generated using unsupervised learning method(s); and (iv) training, by one or more processors, an artificial neural network adapted to generate vector representations for natural language text, where the training is based, at least in part, on the received natural language text, the generated set of first metadata, and the generated set of second metadata.

First claim

What is claimed is:

1. A method comprising:
receiving, by one or more processors, a set of natural language text;
generating, by one or more processors, a set of first metadata for the set of natural language text, where the first metadata is generated using supervised learning method(s);
generating, by one or more processors, a set of second metadata for the set of natural language text, where the second metadata is generated using unsupervised learning method(s);
training, by one or more processors, an artificial neural network adapted to generate vector representations for natural language text, where the training is based, at least in part, on the received natural language text, the generated set of first metadata, and the generated set of second metadata;
generating, by one or more processors, a set of at least two vector representations for the set of natural language text using the trained artificial neural network, where each vector representation of the set of at least two vector representations pertains to a respective subset of natural language text from the set of natural language text;
generating, by one or more processors, a vector representation pertaining to the set of natural language text by adding each of the vector representations in the generated set of at least two vector representations; and
storing, by one or more processors, the generated vector representation pertaining to the set of natural language text for use by a natural language processing system.

2. The method of claim 1 , further comprising:
determining, by one or more processors, an amount of similarity between at least two subsets of natural language text from the set of natural text by comparing their respectively generated vector representations.

3. The method of claim 2 , wherein each of the at least two subsets of natural language text is a word.

4. The method of claim 1 , further comprising:
generating, by one or more processors, a set of first metadata for the generated set of at least two vector representations, where the first metadata for the generated set of at least two vector representations is generated using supervised learning method(s);
generating, by one or more processors, a set of second metadata for the set of at least two vector representations, where the second metadata for the generated set of at least two vector representations is generated using unsupervised learning method(s); and
training, by one or more processors, the artificial neural network based, at least in part, on the generated set of at least two vector representations, the generated set of first metadata for the set of at least two vector representations, and the generated set of second metadata for the set of at least two vector representations.

5. The method of claim 1 , further comprising:
generating, by one or more processors, a set of initial vector representations for the set of natural language text;
generating, by one or more processors, a set of first metadata vector representations for the generated set of first metadata; and
generating, by one or more processors, a set of second metadata vector representations for the generated set of second metadata;
wherein the training of the artificial neural network is further based, at least in part, on the generated set of initial vector representations, the generated set of first metadata vector representations, and the generated set of second metadata vector representations.

6. The method of claim 1 , wherein the supervised learning methods utilize at least one of a natural language processing annotator or an ontology.

7. The method of claim 1 , wherein the unsupervised learning methods are based on at least one of reconstruction error or language modeling.

Assignees

• IBM

Inventors

• Cao Liangliang
• Fan James J
• Wang Chang
• Xiang Bing
• Zhou Bowen

Classifications

• G10L15/16

  using artificial neural networks · CPC title

• G10L15/063 Primary

  Training · CPC title

• G06N20/00

  Machine learning · CPC title

• G06N5/022

  Knowledge engineering; Knowledge acquisition · CPC title

• G06F40/40

  Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) · CPC title