Dynamic threshold filtering for watched questions

Patent metadata

Field	Value
Publication number	US-10565503-B2
Application number	US-201615205077-A
Country	US
Kind code	B2
Filing date	Jul 8, 2016
Priority date	Jul 8, 2016
Publication date	Feb 18, 2020
Grant date	Feb 18, 2020

Abstract

Embodiments are directed to a watched questions threshold filtering system that functions to determine and deliver to a user relevant and significant data changes with respect to a user's goals, as defined by a notification threshold value provided by the user. The user is provided with an option to flag one or more queries for automatic re-querying. Confidence scores are processed on new data (i.e., data ingested after the original question was asked) by utilizing a confidence threshold for indicating if the new data warrants alerting a user.

First claim

We claim:

1. A computer implemented method for identifying new informative responses to answer a question based on new data and user input in a system capable of answering questions, the system comprising a processor and a memory comprising instructions executed by the processor, the method comprising:
receiving as inputs:
a watched question for monitoring;
a notification threshold from a user;
a first set of responses on a first date with a corresponding first set of confidence levels resulting from the watched question; and
a time stamp indicating the first date when the first set of responses were returned;

querying ingested data sources after the first date for answers to the watched question to determine a second set of responses and a second set of confidence levels;
calculating a dynamic threshold based on the second set of confidence levels and the notification threshold, wherein the dynamic threshold is calculated as
Dynamic Threshold=|MaxConf−(ConfDelta·Threshold Weight)|

in which MaxConf is defined as a maximum confidence value of the second set of confidence levels, ConfDelta is defined as the difference between the maximum confidence value and a minimum confidence value of the second set of confidence levels, and the threshold weight is defined as the notification threshold value inputted by the user and comprises a weighting factor from 0 to 1;
comparing the second set of confidence levels to the dynamic threshold to determine qualifying responses from the second set of responses that meet the dynamic threshold; and
responsive to determining the qualifying responses from the second set of responses that meet the dynamic threshold, providing a notification to the user.

2. The method of claim 1 , wherein the second set of responses comprises a set of passages and a set of answers.

3. The method of claim 2 , wherein the notification includes an indication of the at least one of one or more passages and one or more answers that meet the dynamic threshold.

4. The method of claim 2 , wherein calculating the dynamic threshold comprises calculating a first dynamic threshold for the set of passages and a second dynamic threshold for the set of answers.

5. The method of claim 1 , further comprising:
providing a user interface allowing the user to input the watched question and the notification threshold.

6. The method of claim 1 , wherein the notification threshold is adjustable by the user of the system to adjust a quality of the responses from the second set of responses that meet the dynamic threshold.

7. The method of claim 1 , wherein one or more of the responses from the second set of responses that meet the dynamic threshold have a lower associated confidence level than one or more confidence levels from the first set of confidence levels corresponding to one or more responses from the first set of responses.

8. A computer program product for identifying new informative responses to answer a question based on new data and user input in a system capable of answering questions, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:
receive as inputs:
a watched question for monitoring;
a notification threshold from a user;
a first set of responses on a first date with a corresponding first set of confidence levels resulting from the watched question; and
a time stamp indicating the first date when the first set of responses were returned;

query ingested data sources after the first date for answers to the watched question to determine a second set of responses and a second set of confidence levels;
calculate a dynamic threshold based on the second set of confidence levels and the notification threshold, wherein the dynamic threshold is calculated as
Dynamic Threshold=|MaxConf−(ConfDelta Threshold Weight)|

in which MaxConf is defined as a maximum confidence value of the second set of confidence levels, ConfDelta is defined as the difference between the maximum confidence value and a minimum confidence value of the second set of confidence levels, and the threshold weight is defined as the notification threshold value inputted by the user and comprises a weighting factor from 0 to 1;
compare the second set of confidence levels to the dynamic threshold to determine qualifying responses from the second set of responses that meet the dynamic threshold; and
responsive to determining the qualifying responses from the second set of responses that meet the dynamic threshold, provide a notification to the user.

9. The computer program product of claim 8 , wherein the second set of responses comprises a set of passages and a set of answers.

10. The computer program product of claim 9 , wherein the notification includes an indication of the at least one of one or more passages and one or more answers that meet the dynamic threshold.

11. The computer program product of claim 8 , wherein the program instructions executable by the processor further cause the processor to:
provide a user interface allowing the user to input the watched question and the notification threshold.

12. The computer program product of claim 8 , wherein the notification threshold is adjustable by the user of the system to adjust a quality of the responses from the second set of responses that meet the dynamic threshold.

13. The computer program product of claim 8 , wherein one or more of the responses from the second set of responses that meet the dynamic threshold have a lower associated confidence level than one or more confidence levels from the first set of confidence levels corresponding to one or more responses from the first set of responses.

14. A system for identifying new informative responses to answer a question based on new data and user input, the system comprising:
a processor configured to:
receive as inputs:
a watched question for monitoring;
a notification threshold from a user;
a first set of responses on a first date with a corresponding first set of confidence levels resulting from the watched question; and
a time stamp indicating the first date when the first set of responses were returned;

query ingested data sources after the first date for answers to the watched question to determine a second set of responses and a second set of confidence levels;
calculate a dynamic threshold based on the second set of confidence levels and the notification threshold, wherein the dynamic threshold is calculated as
Dynamic Threshold=|MaxConf−(ConfDelta·Threshold Weight)|

in which MaxConf is defined as a maximum confidence value of the second set of confidence levels, ConfDelta is defined as the difference between the maximum confidence value and a minimum confidence value of the second set of confidence levels, and the threshold weight is defined as the notification threshold value inputted by the user and comprises a weighting factor from 0 to 1;
compare the second set of confidence levels to the dynamic threshold to determine responses from the second set of responses that meet the dynamic threshold; and
responsive to determining the qualifying responses from the second set of responses that meet the dynamic threshold, provide a notification to the user.

15. The system of claim 14 , wherein the second set of responses comprises a set of passages and a set of answers.

16. The system of claim 15 , wherein the notification includes an indication of the at least one of one or more passages and one or more answer

Assignees

• IBM

Inventors

• Beller Charles E
• Chase Jr Paul J
• Darden Richard L
• Drzewucki Michael
• Katz Edward G
• Phipps Christopher

Classifications

• G06F16/90335

  Query processing · CPC title

• G06N5/02 Primary

  Knowledge representation; Symbolic representation · CPC title

• G06N5/04 Primary

  Inference or reasoning models · CPC title