Avoiding supporting evidence processing when evidence scoring does not affect final ranking of a candidate answer

Patent metadata

Field	Value
Publication number	US-2016180244-A1
Application number	US-201414576893-A
Country	US
Kind code	A1
Filing date	Dec 19, 2014
Priority date	Dec 19, 2014
Publication date	Jun 23, 2016
Grant date	—

Abstract

Systems and computer program products to provide selective supporting evidence processing by applying a first machine learning (ML) model to a first candidate answer to generate a first confidence score that does not consider supporting evidence for the first candidate answer, determining, from a second ML model, an expected contribution of processing supporting evidence for the first candidate answer, and upon determining that the expected contribution does not exceed a specified threshold, skipping supporting evidence processing for the first candidate answer.

First claim

1 .- 7 . (canceled)

8 . A system, comprising:
one or more computer processors; and a memory containing a program, which when executed by the one or more computer processors, performs an operation to provide selective supporting evidence processing, the operation comprising:
applying a first machine learning (ML) model to a first candidate answer to generate a first confidence score that does not consider supporting evidence for the first candidate answer;
determining, from a second ML model, an expected contribution of processing supporting evidence for the first candidate answer; and
upon determining that the expected contribution does not exceed a specified threshold, skipping supporting evidence processing for the first candidate answer.

9 . The system of claim 8 , wherein the second ML model specifies: (i) a weighted coefficient of processing supporting evidence for the first candidate answer, and (ii) a range of supporting evidence feature scores, wherein each supporting evidence feature score in the range of supporting evidence scores was observed during a training session.

10 . The system of claim 9 , wherein the expected contribution comprises a product of the weighted coefficient and at least one supporting evidence feature score, of the range of supporting evidence feature scores.

11 . The system of claim 8 , wherein the threshold comprises at least one of: (i) a difference between the confidence score of a second candidate answer and the confidence score of the first candidate answer, wherein the confidence score of the second candidate answer is generated by applying the first ML model to the second candidate answer, and (ii) a difference between a confidence score threshold and the first confidence score.

12 . The system of claim 8 , the operation further comprising:
upon determining that the expected contribution exceeds the specified threshold, processing supporting evidence for the first candidate answer; scoring the first candidate answer; and ranking the first candidate answer relative to a set of other candidate answers based on a respective score for each candidate answer.

13 . The system of claim 12 , the operation further comprising:
prior to processing supporting evidence for the first candidate answer, determining that the first confidence score exceeds a minimum confidence threshold.

14 . The system of claim 8 , the operation further comprising:
producing the first and second ML models during a training session.

15 . A computer program product to provide selective supporting evidence processing, comprising:
a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to:
apply a first machine learning (ML) model to a first candidate answer to generate a first confidence score that does not consider supporting evidence for the first candidate answer;
determine, from a second ML model, an expected contribution of processing supporting evidence for the first candidate answer; and
upon determining that the expected contribution does not exceed a specified threshold, skipping supporting evidence processing for the first candidate answer.

16 . The computer program product of claim 15 , wherein the second ML model specifies: (i) a weighted coefficient of processing supporting evidence for the first candidate answer, and (ii) a range of supporting evidence feature scores, wherein each supporting evidence feature score in the range of supporting evidence scores was observed during a training session.

17 . The computer program product of claim 16 , wherein the expected contribution comprises a product of the weighted coefficient and at least one supporting evidence feature score, of the range of supporting evidence feature scores.

18 . The computer program product of claim 15 , wherein the threshold comprises at least one of: (i) a difference between the confidence score of a second candidate answer and the confidence score of the first candidate answer, wherein the confidence score of the second candidate answer is generated by applying the first ML model to the second candidate answer, and (ii) a difference between a confidence score threshold and the first confidence score.

19 . The computer program product of claim 16 , wherein the computer-readable program code is further executable to:
upon determining that the expected contribution exceeds the specified threshold, process supporting evidence for the first candidate answer; score the first candidate answer; and rank the first candidate answer relative to a set of other candidate answers based on a respective score for each candidate answer.

20 . The computer program product of claim 19 , wherein the first and second ML models are produced during a training session, wherein the computer-readable program code is further executable to:
produce the first and second ML models during the training session.

Assignees

• IBM

Inventors

• Britt Michael W
• Lange-Pearson Adam C
• Pitzen Thomas P
• Stevens Richard J

Classifications

• H04L45/08

  Learning-based routing, e.g. using neural networks or artificial intelligence · CPC title

• G06N99/005 Primary

  Physics · mapped topic

• G06F17/30867

  Physics · mapped topic

• G06F17/30663

  Physics · mapped topic

• G06N5/022

  Knowledge engineering; Knowledge acquisition · CPC title