Context based synonym filtering for natural language processing systems

Patent metadata

Field	Value
Publication number	US-9378204-B2
Application number	US-201414285019-A
Country	US
Kind code	B2
Filing date	May 22, 2014
Priority date	May 22, 2014
Publication date	Jun 28, 2016
Grant date	Jun 28, 2016

Abstract

Mechanisms are provided for performing context based synonym filtering for natural language processing. Content is parsed into one or more conceptual units, wherein each conceptual unit comprises a portion of text of the content that is associated with a single concept. For each conceptual unit, a term in the conceptual unit is identified that has a synonym to be utilized during natural language processing of the content. A first measure of relatedness of the term to at least one other term in the conceptual unit is determined. A second measure of relatedness of the synonym of the term to the at least one other term in the conceptual unit is determined. A determination whether or not to utilize the synonym when performing natural language processing on the conceptual unit is made based on the first and second measures of relatedness and natural language processing on the content is performed accordingly.

First claim

What is claimed is:

1. A method, in a data processing system comprising a processor and a memory, for performing context based synonym filtering for natural language processing, the method comprising:
parsing, by the data processing system, content into one or more conceptual units, wherein each conceptual unit comprises a portion of text of the content that is associated with a single concept;
for each conceptual unit in the one or more conceptual units, identifying, by the data processing system, a term in the conceptual unit that has a synonym to be utilized during natural language processing of the content;
determining, by the data processing system, a first measure of relatedness of the term to at least one other term in the conceptual unit;
determining, by the data processing system, a second measure of relatedness of the synonym of the term to the at least one other term in the conceptual unit;
determining, by the data processing system, whether or not to utilize the synonym when performing natural language processing on the conceptual unit, based on the first measure of relatedness and second measure of relatedness; and
performing, by the data processing system, natural language processing on the content based on results of determining whether or not to utilize the synonym, wherein the first measure of relatedness is calculated as the sum, from 1 to N, where N is a number of remaining words in the conceptual unit, of the quantity 1/N*(f(Wn, ORIG)), where f( ) is a semantic distance function, W is the set of remaining words in the conceptual unit, and ORIG is the term, and wherein the second measure of relatedness is calculated as the sum, from 1 to N, of the quantity alpha*1/N*(f(Wn, SYN)), where alpha is a constant, and SYN is the synonym, wherein determining whether or not to utilize the synonym when performing natural language processing on the conceptual unit comprises:
comparing the first measure of relatedness to the second measure of relatedness using the following relationship:

∑

n
=
1

N

⁢

1
N

⁢

f
⁡

(

W
n

,
ORIG

)

≤

∑

n
=
1

N

⁢

α
⁢

1
N

⁢

f
⁡

(

W
n

,
SYN

)

⁢

{

pass

fail

}

;
and

determining to utilize the synonym when performing natural language processing on the conceptual unit in response to the relationship being satisfied.

2. The method of claim 1 , wherein the content is a question input to a question and answer mechanism of the data processing system that implements the natural language processing on the question.

3. The method of claim 1 , wherein determining whether or not to utilize the synonym when performing natural language processing on the conceptual unit comprises:
comparing the first measure of relatedness to the second measure of relatedness; and
determining to utilize the synonym when performing natural language processing on the conceptual unit in response to the first measure having a specified relationship to the second measure.

4. The method of claim 3 , wherein the specified relationship is that the first measure is greater than or equal to the second measure.

5. The method of claim 1 , wherein the first measure of relatedness of the term to at least one other term in the conceptual unit is a first semantic distance between the term and the at least one other term in the conceptual unit, and wherein the second measure of relatedness of the synonym of the term to the at least one other term in the conceptual unit is a second semantic distance between the synonym and the at least one other term in the conceptual unit.

6. The method of claim 5 , wherein the first semantic distance and the second semantic distance are generated using at least one of a Normalized Compressive Distance algorithm, a Normalized Google Distance algorithm, or other semantic similarity algorithm.

7. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:
parse content into one or more conceptual units, wherein each conceptual unit comprises a portion of text of the content that is associated with a single concept;
for each conceptual unit in the one or more conceptual units, identify a term in the conceptual unit that has a synonym to be utilized during natural language processing of the content;
determine a first measure of relatedness of the term to at least one other term in the conceptual unit;
determine a second measure of relatedness of the synonym of the term to the at least one other term in the conceptual unit;
determine whether or not to utilize the synonym when performing natural language processing on the conceptual unit based on the first measure of relatedness and the second measure of relatedness; and
perform natural language processing on the content based on results of determining whether or not to utilize the synonym, wherein the first measure of relatedness is calculated as the sum, from 1 to N, where N is a number of remaining words in the conceptual unit, of the quantity 1/N*(f(Wn, ORIG)), where f( ) is a semantic distance function, W is the set of remaining words in the conceptual unit, and ORIG is the term, and wherein the second measure of relatedness is calculated as the sum, from 1 to N, of the quantity alpha*1/N*(f(Wn, SYN)), where alpha is a constant, and SYN is the synonym, wherein the computer readable program further causes the computing device to determine whether or not to utilize the synonym when p

Assignees

• IBM

Inventors

• Mueller Kay
• Nolan Christopher M
• Visotski William G
• Wilson David E

Classifications

• G06F40/30

  Semantic analysis · CPC title

• G06F16/221

  Column-oriented storage; Management thereof · CPC title

• G06F40/40

  Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) · CPC title

• G06F40/247 Primary

  Thesauruses; Synonyms · CPC title

• G06F16/951

  Indexing; Web crawling techniques · CPC title