System and method for automated analysis of ground truth using confidence model to prioritize correction options

Patent metadata

Field	Value
Publication number	US-11216739-B2
Application number	US-201816044643-A
Country	US
Kind code	B2
Filing date	Jul 25, 2018
Priority date	Jul 25, 2018
Publication date	Jan 4, 2022
Grant date	Jan 4, 2022

Abstract

A method, system and computer-usable medium are disclosed for automated analysis of ground truth using confidence model to prioritize correction options. In certain embodiments, the ground truth data is analyzed to identify review-candidates. A confidence level may be assigned to each of the identified review-candidates and the review-candidates are prioritized, at least in part, using the assigned confidence levels. The review-candidates are electronically presented in prioritized order to solicit verification or correction feedback for updating the ground truth data.

First claim

What is claimed is:

1. A computer-implemented method for automated analysis of ground truth using an information processing system having a processor and a memory, the method comprising:
receiving, by the information processing system, ground truth data;
analyzing, by the information processing system, the ground truth data to identify review-candidates;
assigning, by the information processing system, a confidence level to each of the identified review-candidates;
prioritizing, by the information processing system, the review-candidates based at least on the assigned confidence levels;
electronically presenting, by the information processing system, the review-candidates in prioritized order to solicit corrective feedback for updating the ground truth data;
generating, by the information processing system, suggested fixes for the review-candidates; and
grouping identified review candidates having the same suggested fixes;
electronically presenting the grouped review-candidates in prioritized order along with the suggested fixes to solicit corrective feedback for updating the ground truth data using the suggested fixes; and,
training a question answer (QA) system using the suggested fixes.

2. The computer-implemented method of claim 1 , wherein prioritizing the review-candidates further comprises:
prioritizing a review-candidate based on an impact of changing the review-candidate in the ground truth data using one or more of the respective suggested fixes.

3. The computer-implemented method of claim 2 , wherein
the impact of changing the review-candidate in the ground truth data is based, at least in part, on a number of ground truth data entries that would be changed using the respective suggested fixes.

4. The computer-implemented method of claim 1 , further comprising:
identifying, by the information processing system, review-candidates based on similarities between different attribute names; and
assigning, by the information processing system, a high confidence level to review-candidates having different attribute names within a predetermined edit distance.

5. The computer-implemented method of claim 1 , further comprising:
identifying, by the information processing system, review-candidates based on differences in data types in ground truth entries for a given attribute; and
assigning, by the information processing system, a high confidence level to review-candidates having different data types for the given attribute.

6. A system comprising:
a processor;
a data bus coupled to the processor; and
a non-transitory, computer-readable storage medium embodying computer program code, the non-transitory, computer-readable storage medium being coupled to the data bus, the computer program code interacting with a plurality of computer operations and comprising instructions executable by the processor and configured for:
receiving ground truth data;
analyzing the ground truth data to identify review-candidates;
assigning a confidence level to each of the identified review-candidates;
prioritizing the review-candidates based at least on the assigned confidence levels;
electronically presenting the review-candidates in prioritized order to solicit corrective feedback for updating the ground truth data;
generating, by the information processing system, suggested fixes for the review-candidates; and
grouping identified review candidates having the same suggested fixes;
electronically presenting the grouped review-candidates in prioritized order along with the suggested fixes to solicit corrective feedback for updating the ground truth data using the suggested fixes; and,
training a question answer (QA) system using the suggested fixes.

7. The system of claim 6 , wherein prioritizing the review-candidates further comprises:
prioritizing a review-candidate based on an impact of changing the review-candidate in the ground truth data using one or more of the respective suggested fixes.

8. The system of claim 7 , wherein:
the impact of changing the review-candidate in the ground truth data is based, at least in part, on a number of ground truth data entries that would be changed using the respective suggested fixes.

9. The system of claim 6 , wherein the instructions are further configured for:
identifying review-candidates based on similarities between different attribute names; and
assigning a high confidence level to review-candidates having different attribute names within a predetermined edit distance.

10. The system of claim 6 , wherein the instructions are further configured for:
identifying review-candidates based on differences in data types in ground truth entries for a given attribute; and
assigning a high confidence level to review-candidates having different data types for the given attribute.

11. A non-transitory, computer-readable storage medium embodying computer program code, the computer program code comprising computer executable instructions configured for:
receiving ground truth data;
analyzing the ground truth data to identify review-candidates;
assigning a confidence level to each of the identified review-candidates;
prioritizing the review-candidates based at least on the assigned confidence levels;
electronically presenting the review-candidates in prioritized order to solicit corrective feedback for updating the ground truth data;
generating, by the information processing system, suggested fixes for the review-candidates; and
grouping identified review candidates having the same suggested fixes;
electronically presenting the grouped review-candidates in prioritized order along with the suggested fixes to solicit corrective feedback for updating the ground truth data using the suggested fixes; and,
training a question answer (QA) system using the suggested fixes.

12. The non-transitory, computer-readable storage medium of claim 11 , wherein prioritizing the review-candidates further comprises:
prioritizing a review-candidate based on an impact of changing the review-candidate in the ground truth data using one or more of the respective suggested fixes.

13. The non-transitory, computer-readable storage medium of claim 12 , wherein
the impact of changing the review-candidate in the ground truth data is based, at least in part, on a number of ground truth data entries that would be changed using the respective suggested fixes.

14. The non-transitory, computer-readable storage medium of claim 11 , wherein the instructions are further configured for:
identifying review-candidates based on similarities between different attribute names; and
assigning a high confidence level to review-candidates having different attribute names within a predetermined edit distance.

15. The non-transitory, computer-readable storage medium of claim 11 , wherein the instructions are further configured for:
identifying review-candidates based on differences in data types in ground truth entries for a given attribute; and
assigning a high confidence level to review-candidates having different data types for the given attribute.

Assignees

• IBM

Inventors

• Freed Andrew R
• Christianson Kyle G
• Phipps Christopher

Classifications

• G06N3/042

  Knowledge-based neural networks; Logical representations of neural networks · CPC title

• G06N20/00

  Machine learning · CPC title

• G06N3/006

  based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] · CPC title

• G06N5/048 Primary

  Fuzzy inferencing · CPC title

• G06N5/022 Primary

  Knowledge engineering; Knowledge acquisition · CPC title