Automated correction of natural language processing systems

Patent metadata

Field	Value
Publication number	US-9535894-B2
Application number	US-201514696677-A
Country	US
Kind code	B2
Filing date	Apr 27, 2015
Priority date	Apr 27, 2015
Publication date	Jan 3, 2017
Grant date	Jan 3, 2017

Abstract

Machine logic that automatically detects natural language processing (NLP) system annotation errors and correspondingly updates NLP annotators to prevent future erroneous annotations by performing the following steps: (i) determining that a first annotation error has occurred in an annotation of a corpus by the natural language processing system; (ii) generating a candidate set of annotation correction actions, where each annotation correction action of the set is adapted to prevent an occurrence of an error similar to the first annotation error by the natural language processing system; (iii) selecting an annotation correction action from the candidate set of annotation correction actions, based, at least in part, on a set of annotation correction confidence characteristics; and (iv) automatically applying the selected annotation correction action to the natural language processing system.

First claim

What is claimed is:

1. A method comprising:
causing, by one or more processors, a first natural language processing (NLP) annotator of an NLP system to annotate a corpus, thereby producing an annotated corpus that includes a first set of annotation(s);
causing, by one or more processors, a second NLP annotator of the NLP system to annotate the annotated corpus that includes the first set of annotation(s), thereby producing a second set of annotation(s) that annotate the annotated corpus that includes the first set of annotation(s);
determining, by one or more processors, based, at least in part, on the second set of annotation(s), that a first annotation error has occurred in the annotation of the corpus by the first NLP annotator;
identifying, by one or more processors, a cause of the first annotation error;
generating, by one or more processors, a candidate set of annotation correction actions, where each annotation correction action of the set is directed to the identified cause and is adapted to prevent an occurrence of an error similar to the first annotation error by the first NLP annotator;
selecting, by one or more processors, an annotation correction action from the candidate set of annotation correction actions, based, at least in part, on a set of annotation correction confidence characteristics and based, at least in part, on an impact analysis performed against one or more ground truth annotations, the ground truth annotations having been performed by humans;
automatically applying, by one or more processors, the selected annotation correction action to the first NLP annotator; and
causing, by one or more processors, the first NLP annotator to annotate a new corpus, thereby producing new annotation(s) based on the applied annotation correction action.

2. The method of claim 1 , wherein determining that the first annotation error has occurred is based, at least in part, on metrics collected during the annotation of the corpus by the first NLP annotator.

3. The method of claim 1 , wherein the identified cause of the first annotation error is a missing item.

4. The method of claim 3 , wherein the missing item is a contextual trigger.

5. The method of claim 1 , wherein the annotation correction confidence characteristics include a degree of harmony/discord of a data point.

6. The method of claim 1 , wherein the selected annotation correction action includes adding a set of word(s) to an inclusion list, or adding a set of word(s) to an exception list.

7. The method of claim 1 , where in the selected annotation correction action includes updating a natural language processing rule.

8. A computer program product comprising a computer readable storage medium having stored thereon:
program instructions programmed to cause a first natural language processing (NLP) annotator of an NLP system to annotate a corpus, thereby producing an annotated corpus that includes a first set of annotation(s);
program instructions programmed to cause a second NLP annotator of the NLP system to annotate the annotated corpus that includes the first set of annotation(s), thereby producing a second set of annotation(s) that annotate the annotated corpus that includes the first set of annotation(s);
program instructions programmed to determine, based, at least in part, on the second set of annotation(s), that a first annotation error has occurred in the annotation of the corpus by the first NLP annotator;
program instructions programmed to identify a cause of the first annotation error;
program instructions programmed to generate a candidate set of annotation correction actions, where each annotation correction action of the set is directed to the identified cause and is adapted to prevent an occurrence of an error similar to the first annotation error by the first NLP annotator;
program instructions programmed to select an annotation correction action from the candidate set of annotation correction actions, based, at least in part, on a set of annotation correction confidence characteristics and based, at least in part, on an impact analysis performed against one or more ground truth annotations, the ground truth annotations having been performed by humans; and
program instructions programmed to automatically apply the selected annotation correction action to the first NLP annotator; and
program instructions programmed to cause the first NLP annotator to annotate a new corpus, thereby producing new annotation(s) based on the applied annotation correction action.

9. The computer program product of claim 8 , wherein determining that the first annotation error has occurred is based, at least in part, on metrics collected during the annotation of the corpus by the first NLP annotator.

10. The computer program product of claim 8 , wherein the identified cause of the first annotation error is a missing item in a dictionary.

11. The computer program product of claim 8 , where in the selected annotation correction action includes updating a natural language processing rule.

12. A computer system comprising:
a processor(s) set; and
a computer readable storage medium;
wherein:
the processor set is structured, located, connected and/or programmed to run program instructions stored on the computer readable storage medium; and
the program instructions include:
program instructions programmed to cause a first natural language processing (NLP) annotator of an NLP system to annotate a corpus, thereby producing an annotated corpus that includes a first set of annotation(s);
program instructions programmed to cause a second NLP annotator of the NLP system to annotate the annotated corpus that includes the first set of annotation(s), thereby producing a second set of annotation(s) that annotate the annotated corpus that includes the first set of annotation(s);
program instructions programmed to determine, based, at least in part, on the second set of annotation(s), that a first annotation error has occurred in the annotation of the corpus by the first NLP annotator;
program instructions programmed to identify a cause of the first annotation error;
program instructions programmed to generate a candidate set of annotation correction actions, where each annotation correction action of the set is directed to the identified cause and is adapted to prevent an occurrence of an error similar to the first annotation error by the first NLP annotator;
program instructions programmed to select an annotation correction action from the candidate set of annotation correction actions, based, at least in part, on a set of annotation correction confidence characteristics and based, at least in part, on an impact analysis performed against one or more ground truth annotations, the ground truth annotations having been performed by humans; and
program instructions programmed to automatically apply the selected annotation correction action to the first NLP annotator; and
program instructions programmed to cause the first NLP annotator to annotate a new corpus, thereby producing new annotation(s) based on the applied annotation correction action.

13. The computer system of claim 12 , wherein determining that the first annotation error has occurred is based, at least in part, on metrics collected during the annotation of the corpus by the first NLP annotator.

14. The computer system of claim 12 , wherein:
the identified cause of the first annotation error is a missing item in a dictionary.

15. The computer system of claim 12 , wherein the selected annotation correction action includes updating a natural language processing rule.

Assignees

• IBM

Inventors

• Carrier Scott R
• Mustafi Joy
• Omanwar Anil M
• Polisetty Venkata Sai Avinesh

Classifications

• G06F40/40

  Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) · CPC title

• G06F40/242

  Dictionaries · CPC title

• G06F40/279

  Recognition of textual entities · CPC title

• G06F40/169 Primary

  Annotation, e.g. comment data or footnotes · CPC title

• G06F40/189

  Automatic justification · CPC title