Weak supervised abnormal entity detection

Patent metadata

Field	Value
Publication number	US-11423227-B2
Application number	US-202016789804-A
Country	US
Kind code	B2
Filing date	Feb 13, 2020
Priority date	Feb 13, 2020
Publication date	Aug 23, 2022
Grant date	Aug 23, 2022

Abstract

A mechanism is provided to implement an abnormal entity detection mechanism that facilitates detecting abnormal entities in real-time response systems through weak supervision. For each first intent from an entity labeled workspace that matches a second intent in labeled chat logs, when the entity score associated with each first entity or second entity is above a predefined significance level the first entity or the second entity is recorded. For each first intent from the entity labeled workspace that matches the second intent in the labeled chat logs: responsive to the first entity being recorded and the second entity failing to be recorded, that first entity is removed from the training data as being mistakenly included; or, responsive to the second entity being recorded and the first entity failing to be recorded, that second entity is added as a potential business case to the training data.

First claim

What is claimed is:

1. A method, in a data processing system, for comprising at least one processor and at least one memory, wherein the at least one memory comprises instructions that are executed by the at least one processor to configure the at least one processor to implement an abnormal entity detection mechanism that facilitates detecting abnormal entities in real-time response systems through weak supervision, the method comprising:
for each first intent of one or more first intents from an entity labeled workspace, identifying one or more first entities associated with the first intent and an entity score associated with each first entity;
for each second intent of one or more second intents from labeled chat logs, identifying one or more second entities associated with the second intent and an entity score associated with each second entity;
for each first intent from the entity labeled workspace that matches a second intent in the labeled chat logs, recording the first entity or the second entity in a results data structure when the entity score associated with each first entity or second entity is above a predefined significance level; and
for each first intent from the entity labeled workspace that matches the second intent in the labeled chat logs:
responsive to the first entity being recorded in a results data structure and the second entity failing to be recorded in the results data structure, removing that the first entity from the training data as being mistakenly included in the training data; or
responsive to the second entity being recorded in the results data structure and the first entity failing to be recorded in the results data structure, adding the second entity as a potential business case to the training data.

2. The method of claim 1 , wherein the one or more first entities associated with the first intent in the entity labeled workspace are identified from intent data utilizing weak entity labeling through natural language processing.

3. The method of claim 1 , wherein each second intent is identified from human conversation chat logs utilizing weak intent labeling through natural language processing and wherein the one or more second entities associated with each second intent are identified from human conversation chat logs utilizing weak entity labeling through natural language processing.

4. The method of claim 1 , wherein the entity score associated with each first entity associated with each first intent is generated through correlation analysis of an intent of a sentence identified by a customer to entities identified from the sentence associated with the identified intent.

5. The method of claim 1 , wherein the entity score associated with each second entity associated with each second intent is generated through correlation analysis of a predicted intent of a sentence identified to entities identified from the sentence associated with the predicted intent.

6. The method of claim 1 , further comprising:
performing a pairwise semantic evaluation by creating one or more (unigram/bigram)/entity pairs of each of one or more unigrams and bigrams to each first entity in the one or more first entities associated with the first intent;
generating a first set of phrase embedding vectors for each unigram/bigram and a second set of phrase embedding vectors for each first entity;
determining a similarity score based on a cosine distance between each phrase embedding vector for each unigram/bigram and each phrase embedding vector for each first entity; and
responsive to none of the one or more (unigram/bigram)/entity pairs having similarity score greater than a predetermined similarity score, redefining the first entity in the training data.

7. The method of claim 6 , wherein the one or more unigrams and bigrams are identified by:
for each sentence on intent data, performing natural language processing on the sentence to identify the one or more unigrams or bigrams of the sentence as being associated with the intent identified for the sentence thereby forming a semantic labeled workspace;
performing a comparison of intents identified in labeled chat logs to the intents identified in the semantic labeled workspace; and
recording those one or more unigrams and bigrams associated with the intents in the labeled chat logs that match intents in the semantic labeled workspace.

8. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a data processing system, causes the data processing system to implement an abnormal entity detection mechanism that facilitates detecting abnormal entities in real-time response systems through weak supervision, and further causes the data processing system to:
for each first intent of one or more first intents from an entity labeled workspace, identify one or more first entities associated with the first intent and an entity score associated with each first entity;
for each second intent of one or more second intents from labeled chat logs, identify one or more second entities associated with the second intent and an entity score associated with each second entity;
for each first intent from the entity labeled workspace that matches a second intent in the labeled chat logs, record the first entity or the second entity in a results data structure when the entity score associated with each first entity or second entity is above a predefined significance level; and
for each first intent from the entity labeled workspace that matches the second intent in the labeled chat logs:
responsive to the first entity being recorded in a results data structure and the second entity failing to be recorded in the results data structure, remove that the first entity from the training data as being mistakenly included in the training data; or
responsive to the second entity being recorded in the results data structure and the first entity failing to be recorded in the results data structure, add the second entity as a potential business case to the training data.

9. The computer program product of claim 8 , wherein the one or more first entities associated with the first intent in the entity labeled workspace are identified from intent data utilizing weak entity labeling through natural language processing.

10. The computer program product of claim 8 , wherein each second intent is identified from human conversation chat logs utilizing weak intent labeling through natural language processing and wherein the one or more second entities associated with each second intent are identified from human conversation chat logs utilizing weak entity labeling through natural language processing.

11. The computer program product of claim 8 , wherein the entity score associated with each first entity associated with each first intent is generated through correlation analysis of an intent of a sentence identified by a customer to entities identified from the sentence associated with the identified intent.

12. The computer program product of claim 8 , wherein the entity score associated with each second entity associated with each second intent is generated through correlation analysis of a predicted intent of a sentence identified to entities identified from the sentence associated with the predicted intent.

13. The computer program product of claim 8 , wherein the computer readable program further causes the data processing system to:
perform a pairwise semantic evaluation by creating one or more (unigram/bigram)/entity pairs of each of one or more unigrams and bigrams to each first entity in the one or

Assignees

• IBM

Inventors

• Qi Haode
• Tan Ming
• Yu Yang
• Rao Navneet N
• Kunc Ladislav
• Potdar Saloni

Classifications

• H04L51/02

  using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages · CPC title

• G06N20/20

  Ensemble learning · CPC title

• G06N20/00

  Machine learning · CPC title

• G06F40/35

  Discourse or dialogue representation · CPC title

• G06F40/295 Primary

  Named entity recognition · CPC title