Automatically improving data annotations by processing annotation properties and user feedback

Patent metadata

Field	Value
Publication number	US-12487976-B2
Application number	US-202117494987-A
Country	US
Kind code	B2
Filing date	Oct 6, 2021
Priority date	Oct 6, 2021
Publication date	Dec 2, 2025
Grant date	Dec 2, 2025

Abstract

Methods, systems, and computer program products for automatically improving data annotations by processing annotation properties and user feedback are provided herein. A computer-implemented method includes obtaining data annotation pairs, each comprising an input data annotation in a first format and a corresponding output data annotation in a second format; determining, within at least a portion of the data annotation pairs, one or more non-diffs; identifying, across the at least a portion of data annotation pairs, data annotation properties associated with multiple intents by processing the non-diffs using property-related rules; modifying at least a portion of the data annotation pairs based on the identified data annotation properties; outputting the modified data annotation pairs to at least one user; and generating a final collection of data annotation pairs by processing at least a portion of the modified data annotation pairs and user feedback received in response to the outputting.

First claim

What is claimed is:

1 . A computer-implemented method comprising:
obtaining a set of data annotation pairs, wherein each of the data annotation pairs comprises an input data annotation in a first format and a corresponding output data annotation in a second format; determining, within at least a portion of the data annotation pairs, one or more non-diffs; determining, across the at least a portion of the data annotation pairs, one or more data annotation properties associated with multiple intents by processing at least a portion of the one or more non-diffs using one or more regular expression learning-based clustering algorithms to group instances of the one or more non-diffs within the at least a portion of the data annotation pairs on a basis of at least one of (i) one or more repeating characters within the one or more non-diffs, (ii) non-diff positioning, and (iii) one or more matching words within the one or more non-diffs; modifying at least a portion of the data annotation pairs based at least in part on the one or more identified data annotation properties; outputting the modified data annotation pairs to at least one user; and generating a final collection of data annotation pairs by processing at least a portion of the modified data annotation pairs and user feedback received in response to the outputting of the modified data annotation pairs; wherein the method is carried out by at least one computing device.

2 . The computer-implemented method of claim 1 , wherein determining the one or more non-diffs comprises processing the at least a portion of the data annotation pairs for presence of similar length non-diffs in respective input data annotations and corresponding output data annotations.

3 . The computer-implemented method of claim 1 , wherein determining the one or more non-diffs comprises processing the at least a portion of the data annotation pairs for presence of identical position placement of at least one non-diff in respective input data annotations and corresponding output data annotations.

4 . The computer-implemented method of claim 1 , wherein determining the one or more non-diffs comprises processing the at least a portion of the data annotation pairs for presence of at least one non-diff of a same token type in respective input data annotations and corresponding output data annotations.

5 . The computer-implemented method of claim 1 , wherein determining the one or more non-diffs comprises processing the at least a portion of the data annotation pairs for presence of one or more repeating characters within at least one non-diff in respective input data annotations and corresponding output data annotations.

6 . The computer-implemented method of claim 1 , wherein the user feedback comprises at least one of acceptance of at least a portion of the one or more identified data annotation properties and rejection of at least a portion of the one or more identified data annotation properties.

7 . The computer-implemented method of claim 1 , wherein modifying the at least a portion of the data annotation pairs comprises generating one or more new data annotation pairs based at least in part on the one or more identified data annotation properties and the obtained set of data annotation pairs.

8 . The computer-implemented method of claim 1 , wherein modifying the at least a portion of the data annotation pairs comprises updating one or more of the obtained set of data annotation pairs based at least in part on the one or more identified data annotation properties.

9 . The computer-implemented method of claim 1 , further comprising:
computing quality scores for the obtained set of data annotation pairs, wherein the quality scores are based at least in part on extent of multiple intents associated with the obtained set of data annotation pairs.

10 . The computer-implemented method of claim 9 , further comprising:
computing quality scores for the final collection of data annotation pairs, wherein the quality scores are based at least in part on extent of multiple intents associated with the final collection of data annotation pairs; and comparing the quality scores for the final collection of data annotation pairs to the quality scores for the obtained set of data annotation pairs.

11 . The computer-implemented method of claim 1 , wherein software implementing the method is provided as a service in a cloud environment.

12 . A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computing device to cause the computing device to:
obtain a set of data annotation pairs, wherein each of the data annotation pairs comprises an input data annotation in a first format and a corresponding output data annotation in a second format; determine, within at least a portion of the data annotation pairs, one or more non-diffs; determine, across the at least a portion of the data annotation pairs, one or more data annotation properties associated with multiple intents by processing at least a portion of the one or more non-diffs using one or more regular expression learning-based clustering algorithms to group instances of the one or more non-diffs within the at least a portion of the data annotation pairs on a basis of at least one of (i) one or more repeating characters within the one or more non-diffs, (ii) non-diff positioning, and (iii) one or more matching words within the one or more non-diffs; modify at least a portion of the data annotation pairs based at least in part on the one or more identified data annotation properties; output the modified data annotation pairs to at least one user; and generate a final collection of data annotation pairs by processing at least a portion of the modified data annotation pairs and user feedback received in response to the outputting of the modified data annotation pairs.

13 . The computer program product of claim 12 , wherein determining the one or more non-diffs comprises processing the at least a portion of the data annotation pairs for presence of similar length non-diffs in respective input data annotations and corresponding output data annotations.

14 . The computer program product of claim 12 , wherein determining the one or more non-diffs comprises processing the at least a portion of the data annotation pairs for presence of identical position placement of at least one non-diff in respective input data annotations and corresponding output data annotations.

15 . The computer program product of claim 12 , wherein determining the one or more non-diffs comprises processing the at least a portion of the data annotation pairs for presence of at least one non-diff of a same token type in respective input data annotations and corresponding output data annotations.

16 . The computer program product of claim 12 , wherein determining the one or more non-diffs comprises processing the at least a portion of the data annotation pairs for presence of one or more repeating characters within at least one non-diff in respective input data annotations and corresponding output data annotations.

17 . The computer program product of claim 12 , wherein the user feedback comprises at least one of acceptance of at least a portion of the one or more identified data annotation properties and rejection of at least a portion of the one or more identified data annotation properties.

18 . The computer program product of claim 12 , wherein the program instructions executable by a comput

Assignees

• IBM

Inventors

• Guttula Shanmukha Chaitanya
• Gupta Nitin
• Lohia Pranay Kumar
• Patel Hima

Classifications

• G06F16/2365

  Ensuring data consistency and integrity · CPC title

• G06F16/215 Primary

  Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors · CPC title