Generating natural language text sentences as test cases for nlp annotators with combinatorial test design

Patent metadata

Field	Value
Publication number	US-2016170972-A1
Application number	US-201414572691-A
Country	US
Kind code	A1
Filing date	Dec 16, 2014
Priority date	Dec 16, 2014
Publication date	Jun 16, 2016
Grant date	—

Abstract

Test cases for a text annotator are generated by determining types of inputs to the annotator and analyzing language structures in a corpus to identify sentence types and grammar constructs. An input type can correspond to multiple grammar constructs. Test cases are generated by performing grammar tree transformations on selected fragments from the corpus based on the sentence types and the grammar constructs. Additional test cases are generated by replacing starting phrases in selected fragments with substitute phrases from dictionaries associated with the input types (a dictionary can include a false synonym for an input type for purposes of negative testing). The two generating approaches can be combined, i.e., performing one or more successive (different) grammar tree transformations to yield a sentence which is then subjected to phrase substitution.

First claim

What is claimed is:

1 . A method of generating test cases for a text annotator comprising:
receiving a corpus of text fragments and a description of the text annotator, by executing first instructions in a computer system; determining types of inputs to the text annotator from the description, by executing second instructions in the computer system; analyzing language structures in the corpus to identify sentence types and grammar constructs, by executing third instructions in the computer system; and generating a first test case by performing a grammar tree transformation on a first selected fragment of the corpus based on the sentence types and the grammar constructs, by executing fourth instructions in the computer system.

2 . The method of claim 1 wherein the first test case is generated by performing a sequence of different successive grammar tree transformations starting with the first selected fragment.

3 . The method of claim 1 further comprising generating a second test case by replacing at least one starting phrase in a second selected fragment of the corpus with a substitute phrase from at least one dictionary associated with one of the types of inputs that corresponds to the starting phrase.

4 . The method of claim 3 wherein the second test case is generated by replacing multiple starting phrases in the second selected fragment with respective substitute phrases from multiple dictionaries associated with different ones of the types of inputs that correspond to the multiple starting phrases.

5 . The method of claim 1 wherein at least one of the types of inputs corresponds to multiple grammar constructs.

6 . The method of claim 1 wherein the dictionary includes a false synonym for the one input type that corresponds to the starting phrase.

7 . The method of claim 1 wherein:
said analyzing includes performing a slot grammar parse of the corpus to determine various parse trees of the corpus including a most common parse tree; and
the first selected fragment is selected in response to a selection bias towards a sentence type which corresponds to the most common parse tree of the corpus.

8 . A computer system comprising:
one or more processors which process program instructions; a memory device connected to said one or more processors; and program instructions residing in said memory device for generating test cases for a text annotator by receiving a corpus of text fragments and a description of the text annotator, determining types of inputs to the text annotator from the description, analyzing language structures in the corpus to identify sentence types and grammar constructs, and generating a first test case by performing a grammar tree transformation on a first selected fragment of the corpus based on the sentence types and the grammar constructs.

9 . The computer system of claim 8 wherein the first test case is generated by performing a sequence of different successive grammar tree transformations starting with the first selected fragment.

10 . The computer system of claim 8 wherein said program instructions further include generating a second test case by replacing at least one starting phrase in a second selected fragment of the corpus with a substitute phrase from at least one dictionary associated with one of the types of inputs that corresponds to the starting phrase.

11 . The computer system of claim 10 wherein the second test case is generated by replacing multiple starting phrases in the second selected fragment with respective substitute phrases from multiple dictionaries associated with different ones of the types of inputs that correspond to the multiple starting phrases.

12 . The computer system of claim 8 wherein at least one of the types of inputs corresponds to multiple grammar constructs.

13 . The computer system of claim 8 wherein the dictionary includes a false synonym for the one input type that corresponds to the starting phrase.

14 . The computer system of claim 8 wherein:
the analyzing includes performing a slot grammar parse of the corpus to determine various parse trees of the corpus including a most common parse tree; and
the first selected fragment is selected in response to a selection bias towards a sentence type which corresponds to the most common parse tree of the corpus.

15 . A computer program product comprising:
a computer readable storage medium; and program instructions residing in said storage medium for generating test cases for a text annotator by receiving a corpus of text fragments and a description of the text annotator, determining types of inputs to the text annotator from the description, analyzing language structures in the corpus to identify sentence types and grammar constructs, and generating a first test case by performing a grammar tree transformation on a first selected fragment of the corpus based on the sentence types and the grammar constructs.

16 . The computer program product of claim 15 wherein the first test case is generated by performing a sequence of different successive grammar tree transformations starting with the first selected fragment.

17 . The computer program product of claim 15 wherein said program instructions further include generating a second test case by replacing at least one starting phrase in a second selected fragment of the corpus with a substitute phrase from at least one dictionary associated with one of the types of inputs that corresponds to the starting phrase.

18 . The computer program product of claim 17 wherein the second test case is generated by replacing multiple starting phrases in the second selected fragment with respective substitute phrases from multiple dictionaries associated with different ones of the types of inputs that correspond to the multiple starting phrases.

19 . The computer program product of claim 15 wherein at least one of the types of inputs corresponds to multiple grammar constructs.

20 . The computer program product of claim 15 wherein the dictionary includes a false synonym for the one input type that corresponds to the starting phrase.

21 . The computer program product of claim 15 wherein:
the analyzing includes performing a slot grammar parse of the corpus to determine various parse trees of the corpus including a most common parse tree; and
the first selected fragment is selected in response to a selection bias towards a sentence type which corresponds to the most common parse tree of the corpus.

Assignees

• IBM

Inventors

• Andrejko Pamela D
• Freed Andrew R
• Murch Cynthia M
• Nielsen Robert L
• Nordland Jan M
• Rivero Humberto R

Classifications

• G06F40/154

  Tree transformation for tree-structured or markup documents, e.g. XSLT, XSL-FO or stylesheets · CPC title

• G06F40/289

  Phrasal analysis, e.g. finite state techniques or chunking · CPC title

• G06F40/56

  Natural language generation · CPC title

• G06F40/253 Primary

  Grammatical analysis; Style critique · CPC title

• G16H15/00

  ICT specially adapted for medical reports, e.g. generation or transmission thereof · CPC title