System and method for determining quality metrics for a question set

Patent metadata

Field	Value
Publication number	US-10528453-B2
Application number	US-201615054393-A
Country	US
Kind code	B2
Filing date	Feb 26, 2016
Priority date	Jan 20, 2016
Publication date	Jan 7, 2020
Grant date	Jan 7, 2020

Abstract

A computer-implemented method is provided for determining quality metrics for a question set. In an implementation, a test question set model may be produced based upon calculated quality metrics of a test question set with respect to a test corpus, and including features representing quality metrics. The test question set model may be compared to a baseline question set model based on a distance calculated between one or more projected model features of the baseline question set model and one or more runtime model features of the test question set model. Contents of the test question set may be adjusted based upon the calculated distance.

First claim

What is claimed is:

1. A computer-implemented method comprising:
producing, by a processor, a test question set model based upon, at least in part, calculated quality metrics of a test question set with respect to a test corpus, and including a plurality of test question set model features representing quality metrics for the test question set in the test question set model to define coverage between the test question set and the test corpus based on one or more possible candidate answers to the test question set identified from the test corpus;
identifying a level of coverage for the test question set by dividing a total number of unique identifiers selected from the test corpus for the test question set by the total number of unique identifiers from the test corpus in producing the test question set model;
identifying a level of non-coverage for the test question set by dividing a number of unique identifiers remaining that were not selected from the test corpus for the test question set by the total number of unique identifiers from the test corpus in producing the test question set model;
comparing, by the processor, the test question set model to a baseline question set model based on calculating a distance between one or more projected model features of the baseline question set model and one or more runtime model features of the test question set model, wherein the test question set model and the baseline question set model each comprise vectors and the distance comprises a vector distance difference;
adjusting, by the processor, contents of the test question set based upon, at least in part, the calculated distance between the projected model features of the baseline question set model and the runtime model features of the test question set model; and
testing a question answering computer system based on the adjusted contents of the test question set.

2. The computer-implemented method of claim 1 , wherein the baseline question set model is produced based on calculated quality metrics of a baseline question set with respect to a baseline corpus and includes a plurality of baseline question set model features representing quality metrics for the baseline question set.

3. The computer-implemented method of claim 2 , wherein the baseline question set model is selected based upon, at least in part, a domain distance between the baseline corpus and the test corpus as a graph difference between domains of the baseline corpus and the test corpus.

4. The computer-implemented method of claim 1 , further including:
projecting the test question set accuracy from the runtime model features of the baseline question set by analyzing the distance between the baseline question set model and the test question set model.

5. The computer-implemented method of claim 1 , further including:
tuning the test question set model by rewarding prominent features of the test question set and penalizing less prominent features of the test question set.

Assignees

• IBM

Inventors

• Kochura Nadiya
• Raghavan Vinodkumar
• Randall Jr Donald H
• Reedy Derek M
• Snow Timothy B

Classifications

• G06N5/04

  Inference or reasoning models · CPC title

• G06N3/006

  based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] · CPC title

• G06F11/3684 Primary

  for test design, e.g. generating new test cases · CPC title

• G06Q10/0639

  Performance analysis of employees; Performance analysis of enterprise or organisation operations · CPC title

• G06F11/3616

  using software metrics · CPC title