Context based synonym filtering for natural language processing systems

Patent metadata

Field	Value
Publication number	US-9792280-B2
Application number	US-201615172577-A
Country	US
Kind code	B2
Filing date	Jun 3, 2016
Priority date	May 22, 2014
Publication date	Oct 17, 2017
Grant date	Oct 17, 2017

Abstract

Mechanisms are provided for performing context based synonym filtering for natural language processing. Content is parsed into one or more conceptual units, wherein each conceptual unit comprises a portion of text of the content that is associated with a single concept. For each conceptual unit, a term in the conceptual unit is identified that has a synonym to be utilized during natural language processing of the content. A first measure of relatedness of the term to at least one other term in the conceptual unit is determined. A second measure of relatedness of the synonym of the term to the at least one other term in the conceptual unit is determined. A determination whether or not to utilize the synonym when performing natural language processing on the conceptual unit is made based on the first and second measures of relatedness and natural language processing on the content is performed accordingly.

First claim

What is claimed is:

1. A method, in a data processing system comprising a processor and a memory, wherein the memory comprises instructions which are executed by the processor to specifically configure the processor to implement a conceptual context synonym verification engine that performs context based synonym filtering for natural language processing, the method comprising:
parsing, by the conceptual context synonym verification engine of the data processing system, content provided by a natural language processing system, into one or more conceptual units, wherein each conceptual unit comprises a portion of text of the content that is associated with a single concept;
for each conceptual unit in the one or more conceptual units, identifying, by the conceptual context synonym verification engine of the data processing system, a term in the conceptual unit that has a synonym to be utilized during natural language processing of the content;
determining, by the conceptual context synonym verification engine of the data processing system, a first measure of relatedness of the term to at least one other term in the conceptual unit;
determining, by the conceptual context synonym verification engine of the data processing system, a second measure of relatedness of the synonym of the term to the at least one other term in the conceptual unit;
determining, by the conceptual context synonym verification engine of the data processing system, whether or not to utilize the synonym when performing natural language processing on the conceptual unit, based on the first measure of relatedness and second measure of relatedness;
configuring, by the conceptual context synonym verification engine, the natural language processing system to utilize or not utilize the synonym, when performing natural language processing on the content, based on results of determining whether or not to utilize the synonym when performing natural language processing on the conceptual unit; and
performing, by the natural language processing system, natural language processing on the content based on results of determining whether or not to utilize the synonym, wherein the content is electronic content that is input to the data processing system.

2. The method of claim 1 , wherein the content is a question input to a question and answer mechanism of the data processing system that implements the natural language processing on the question.

3. The method of claim 1 , wherein determining whether or not to utilize the synonym when performing natural language processing on the conceptual unit comprises:
comparing the first measure of relatedness to the second measure of relatedness; and
determining to utilize the synonym when performing natural language processing on the conceptual unit in response to the first measure having a specified relationship to the second measure.

4. The method of claim 3 , wherein the specified relationship is that the first measure is greater than or equal to the second measure.

5. The method of claim 1 , wherein the first measure of relatedness of the term to at least one other term in the conceptual unit is a first semantic distance between the term and the at least one other term in the conceptual unit, and wherein the second measure of relatedness of the synonym of the term to the at least one other term in the conceptual unit is a second semantic distance between the synonym and the at least one other term in the conceptual unit.

6. The method of claim 5 , wherein the first semantic distance and the second semantic distance are generated using at least one of a Normalized Compressive Distance algorithm, a Normalized Google Distance algorithm, or other semantic similarity algorithm.

7. The method of claim 1 , wherein the first measure of relatedness is calculated as the sum, from 1 to N, where N is a number of remaining words in the conceptual unit, of the quantity 1/N*(f(Wn, ORIG)), where f( ) is a semantic distance function, W is the set of remaining words in the conceptual unit, and ORIG is the term, and wherein the second measure of relatedness is calculated as the sum, from 1 to N, of the quantity alpha*1/N*(f(Wn, SYN)), where alpha is a constant, and SYN is the synonym.

8. The method of claim 7 , wherein determining whether or not to utilize the synonym when performing natural language processing on the conceptual unit comprises:
comparing the first measure of relatedness to the second measure of relatedness using the following relationship:

∑

n
=
1

N

⁢

⁢

1
N

⁢

f
⁡

(

W

n
′

⁢
ORIG

)

≤

∑

n
=
1

N

⁢

⁢

α
⁢

1
N

⁢

f
⁡

(

W

n
′

⁢
SYN

)

⁢

{

pass

fail

}

;

and
determining to utilize the synonym when performing natural language processing on the conceptual unit in response to the relationship being satisfied.

9. A computer program product comprising a non-transitory computer readable medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to be specifically configured to implement a conceptual context synonym verification engine and to:
parse, by the conceptual

Assignees

• IBM

Inventors

• Mueller Kay
• Nolan Christopher M
• Visotski William G
• Wilson David E

Classifications

• G06F16/951

  Indexing; Web crawling techniques · CPC title

• G06F16/335

  Filtering based on additional data, e.g. user or group profiles (filtering in web context G06F16/9535, G06F16/9536) · CPC title

• G06F16/221

  Column-oriented storage; Management thereof · CPC title

• G06F40/205

  Parsing · CPC title

• G06F16/3344

  using natural language analysis · CPC title