Producing insight information from tables using natural language processing

Patent metadata

Field	Value
Publication number	US-9286290-B2
Application number	US-201414262288-A
Country	US
Kind code	B2
Filing date	Apr 25, 2014
Priority date	Apr 25, 2014
Publication date	Mar 15, 2016
Grant date	Mar 15, 2016

Abstract

Mechanisms for generating insight statements from table data are provided. A portion of content comprising a table data structure and text associated with the table data structure is received and at least one of key terms or semantic relationships in the table data structure and the associated text are identified. Fields of an insight statement template are populated with information obtained from the key terms and semantic relationships to generate an insight statement data structure. The insight statement data structure is then output. The insight statement data structure is a natural language statement describing an aspect of the table data structure.

First claim

What is claimed is:

1. A method, in a data processing system, for generating insight statements from table data, the method comprising:
receiving, by the data processing system, a portion of content comprising a table data structure and text associated with the table data structure;
identifying, by the data processing system, at least one of key terms or semantic relationships in the table data structure and the associated text;
identifying, by the data processing system, insight data points in the table data structure based on the identification of at least one of key terms or semantic relationships in the table data structure and the associated text, wherein the insight data points are data points in the table data structure meeting an insight condition set forth in a predefined insight data point rule;
generating, by the data processing system, an insight data structure specifying a field in the table data structure with which the insight data point is associated, an insight condition met by the insight data point, a location in the table data structure corresponding to the insight data point, and a value corresponding to the insight data point;
populating, by the data processing system, fields of an insight statement template with information obtained from the key terms and semantic relationships, based on the insight data structure, to generate an insight statement data structure; and
outputting, by the data processing system, the insight statement data structure, wherein the insight statement data structure is a natural language statement describing an aspect of the table data structure, wherein identifying an insight data point in the table data structure comprises:
performing a matching operation that matches key terms and semantic relationships in the associated text with key terms and semantic relationships in the table data structure to identify matching portions of the table data structure, wherein the identification of the insight data point and generation of the insight data structure is performed for the matching portions of the table data structure.

2. The method of claim 1 , wherein identifying key terms and semantic relationships in the table data structure and the associated text comprises:
performing natural language processing on the table data structure to identify key terms, and semantic relationships associated with key terms, in at least one of columns or rows of the table data structure; and
performing natural language processing on the associated text to identify key terms and semantic relationships associated with key terms in the associated text.

3. The method of claim 1 , wherein the aspect of the table data structure specified in the insight statement is an aspect of the table data not explicitly specified in the table data structure but able to be extracted from the table data through application of insight analysis logic on the table data of the table data structure.

4. The method of claim 1 , wherein key terms and semantic relationships in the associated text are weighted according to a proximity of the key terms and semantic relationships to the table data structure in the content.

5. The method of claim 1 , wherein the text is one of a table summary text associated with the table data structure or a predetermined amount of text in close proximity to the table data structure in the content.

6. The method of claim 1 , wherein the insight statement is output to a natural language processing mechanism which processes the insight statement in association with the table data structure.

7. The method of claim 6 , wherein the natural language processing mechanism is part of a question and answer system, and wherein the question and answer system performs natural language processing on the insight statement when generating candidate answers to an input question.

8. The method of claim 1 , wherein the insight statement is output as one of an annotation of the table data structure, metadata associated with the content, or a separate file linked to the content.

9. The method of claim 1 , wherein the insight data point rules comprise a rule for determining derived information from the information present in the table data structure, and wherein the derived information comprises at least one of a maximum value for a portion of the table data structure, a variance for values in the portion of the table data structure, a distribution of values in the portion of the table data structure, an equivalence for values in the portion of the table data structure, a difference for values in the portion of the table data structure, and a total of values in the portion of the table data structure.

10. The method of claim 1 , wherein the at least one of key terms or semantic relationships in the table data structure and the associated text are key terms and semantic relationships specified in a knowledge base specifically configured for a pre-defined domain.

11. A computer program product comprising a non-transitory computer readable medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:
receive a portion of content comprising a table data structure and text associated with the table data structure;
identify at least one of key terms or semantic relationships in the table data structure and the associated text;
identify, by the data processing system, insight data points in the table data structure based on the identification of at least one of key terms or semantic relationships in the table data structure and the associated text, wherein the insight data points are data points in the table data structure meeting an insight condition set forth in a predefined insight data point rule;
generate, by the data processing system, an insight data structure specifying a field in the table data structure with which the insight data point is associated, an insight condition met by the insight data point, a location in the table data structure corresponding to the insight data point, and a value corresponding to the insight data point;
populate fields of an insight statement template with information obtained from the key terms and semantic relationships, based on the insight data structure, to generate an insight statement data structure; and
output the insight statement data structure, wherein the insight statement data structure is a natural language statement describing an aspect of the table data structure, wherein the computer readable program further causes the computing device to identify an insight data point in the table data structure at least by:
performing a matching operation that matches key terms and semantic relationships in the associated text with key terms and semantic relationships in the table data structure to identify matching portions of the table data structure, wherein the identification of the insight data point and generation of the insight data structure is performed for the matching portions of the table data structure.

12. The computer program product of claim 11 , wherein the computer readable program further causes the computing device to identify key terms and semantic relationships in the table data structure and the associated text at least by:
performing natural language processing on the table data structure to identify key terms, and semantic relationships associated with key terms, in at least one of columns or rows of the table data structure; and
performing natural language processing on the associated text to identify key terms and semantic relationships associated with key terms in the associated text.

Assignees

• IBM

Inventors

• Allen Corville O
• Freed Andrew R

Classifications

• G06F16/245 Primary

  Query processing · CPC title

• G06F40/177

  of tables; using ruled lines · CPC title

• G06F40/30 Primary

  Semantic analysis · CPC title

• G06F40/205

  Parsing · CPC title

• G06F17/2705

  Physics · mapped topic