Unintended bias detection in conversational agent platforms with machine learning model

Patent metadata

Field	Value
Publication number	US-11270080-B2
Application number	US-202016743661-A
Country	US
Kind code	B2
Filing date	Jan 15, 2020
Priority date	Jan 15, 2020
Publication date	Mar 8, 2022
Grant date	Mar 8, 2022

Abstract

A mechanism is provided for implementing a bias detection mechanism that mitigates unintended bias in a conversational agent by leveraging conversational agent definitions, a conversational agent chat logs, and user satisfaction statistics. One or more protected attributes are identified within an utterance from the conversational agent chat logs. Using the identified protected attributes, a replacement utterance with a replacement term is generated for at least one of the identified protected attributes in the utterance. A score is generated for the utterance and the replacement utterance using utterance level relative term importance for protected attributes and regular terms in the utterance and the replacement utterance. Utilizing the scoring, a determination is made as to whether unintended bias exists within the utterance. Responsive to unintended bias being detected, an action is implemented that causes a change to a machine learning model used by the conversational agent.

First claim

What is claimed is:

1. A method, in a data processing system, for comprising at least one processor and at least one memory, wherein the at least one memory comprises instructions that are executed by the at least one processor to configure the at least one processor to implement a bias detection mechanism that mitigates unintended bias in a conversational agent, the method comprising:
identifying one or more protected attributes within a set of original utterances from the conversational agent chat logs;
for each utterance in the set of original utterances, generating a replacement utterance with a replacement term for at least one of the identified protected attributes in the utterance to form a set of replacement utterances;
determining a first subset of utterances where protected attributes exist and result in change in intents between the set of original utterances and the set of replacement utterances:
determining a second subset of utterances from the first subset of utterances where protected attributes exist and result in change in confidences between the set of original utterances and the set of replacement utterances; and
responsive to the change in confidences being above a predetermined threshold, implementing an action that causes a change to a machine learning model used by the conversational agent.

2. The method of claim 1 , wherein identifying the one or more protected attributes comprises:
using regular expression matching or dictionary matching to identify commonly defined protected attribute types.

3. The method of claim 1 , wherein identifying the one or more protected attributes comprises:
using machine learning based Named Entity Recognition (NER).

4. The method of claim 1 , wherein generating the replacement utterance with the replacement term for the at least one of the identified protected attributes comprises:
generating the replacement term that replaces the at least one identified protected attribute with a common equivalent.

5. The method of claim 1 , wherein generating the replacement utterance with the replacement term for the at least one of the identified protected attributes comprises:
generating the replacement term that replaces the at least one identified protected attribute with a random equivalent from a same identity type.

6. The method of claim 1 , further comprising:
determining a percentage of utterances within the conversation agent chat logs that have protected attributes that are relatively important, wherein determining the percentage of utterances within the conversation agent chat logs that have protected attributes that are relatively important comprises:
detecting a prevalence of utterances with protected attributes for a specific conversational agent;
for those utterances with protected attributes associated with the specific conversational agent, detecting a prevalence of utterances with protected attributes with relative importance above a predetermined threshold; and
responsive to detecting a subset of utterances with a relative importance above the predetermined threshold, identifying unintended bias within the subset of utterances.

7. The method of claim 1 , further comprising:
aggregating utterances with protected attributes detected for the conversational agent with relative term importance analysis both with and without protected attribute replacement.

8. The method of claim 1 , further comprising:
segmenting utterances where protected attributes are detected from utterances Where protected attributes are not detected for the conversational agent;
calculating user satisfaction rates on chat sessions where protected attributes are detected versus chat sessions where protected attributes are not detected for the conversational agent;
calculating user satisfaction rates on chat sessions where protected attributes are detected and are considered relatively important versus chat sessions where protected attributes are not detected or chat sessions where protected attributes are detected but are not considered relatively important as determined by the protected attribute analyzer; and
responsive to determining a drop in satisfaction being greater than or equal to a predetermined threshold, identifying unintended bias.

9. The method of claim 1 , further comprising:
generating a score for the utterance and the replacement utterance using utterance level relative term importance for protected attributes and regular terms in the utterance and the replacement utterance:
aggregating statistics from the scoring for the utterance and the replacement utterance; and
responsive to statistics associated with the utterance resulting in a significant score change between the utterance and the replacement utterance, identifying unintended bias.

10. The method of claim 1 , further comprising:
utilizing a set of weights for the protected attributes in the utterance, re-weighing the protected attributes determined to be relatively more important for the conversational agent definition; and
responsive to determining the re-weighing change improves protected attribute detection, identifying unintended bias.

11. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a data processing system, causes the data processing system to implement a bias detection mechanism that mitigates unintended bias in a conversational agent, wherein the computer readable program further causes the data processing system to:
identify one or more protected attributes within a set of original utterances from the conversational agent chat logs;
for each utterance in the set of original utterances, generate a replacement utterance with a replacement term for at least one of the identified protected attributes in the utterance to form a set of replacement utterances;
determine a first subset of utterances where protected attributes exist and result in change in intents between the set of original utterances and the set of replacement utterances;
determine a second subset of utterances from the first subset of utterances where protected attributes exist and result in change in confidences between the set of original utterances and the set of replacement utterances; and
responsive to the change in confidences being above a predetermined threshold, implement an action that causes a change to a machine learning model used by the conversational agent.

12. The computer program product of claim 11 , wherein the computer readable program further causes the data processing system to:
use regular expression matching or dictionary matching to identify commonly defined protected attribute types; or
use machine learning based Named Entity Recognition (NER).

13. The computer program product of claim 11 , wherein the computer readable program further causes the data processing system to:
generate the replacement term that replaces the at least one identified protected attribute with a common equivalent.

14. The computer program product of claim 11 , wherein the computer readable program further causes the data processing system to:
generate the replacement term that replaces the at least one identified protected attribute with a random equivalent from a same identity type.

15. The computer program product of claim 11 , wherein the computer readable program further causes the data processing system to:
determine a percentage of utterances within the conversation agent chat logs that have protected attributes that are relatively important, wherein the comp

Assignees

• IBM

Inventors

• Rao Navneet N
• Tan Ming
• Qi Haode
• Yu Yang
• Karagiannis Panos
• Potdar Saloni

Classifications

• G06F40/30 Primary

  Semantic analysis · CPC title

• G06F40/35 Primary

  Discourse or dialogue representation · CPC title

• G06F16/90332

  Natural language query formulation or dialogue systems · CPC title

• G06F40/289

  Phrasal analysis, e.g. finite state techniques or chunking · CPC title

• G06F16/9032

  Query formulation · CPC title