Swarm fair deep reinforcement learning

Patent metadata

Field	Value
Publication number	US-11416743-B2
Application number	US-201916395187-A
Country	US
Kind code	B2
Filing date	Apr 25, 2019
Priority date	Apr 25, 2019
Publication date	Aug 16, 2022
Grant date	Aug 16, 2022

Abstract

Fair deep reinforcement learning is provided. A microstate of an environment and reaction of items in a plurality of microstates within the environment are observed after an agent performs an action in the environment. Semi-supervised training is utilized to determine bias weights corresponding to the action for the microstate of the environment and the reaction of the items in the plurality of microstates within the environment. The bias weights from the semi-supervised training are merged with non-bias weights using an artificial neural network. Over time, it is determined where bias is occurring in the semi-supervised training based on merging the bias weights with the non-bias weights in the artificial neural network. A deep reinforcement learning model that decreases reliance on the bias weights is generated based on determined bias to increase fairness.

First claim

What is claimed is:

1. A computer-implemented method for providing fair deep reinforcement learning, the computer-implemented method comprising:
receiving, by a computer, via a network, multimedia data capturing a robot performing an action to accomplish a task in a physical environment from a first set of sensors located in the physical environment;
performing, by the computer, using an artificial neural network, an analysis of the multimedia data capturing the robot performing the action to accomplish the task in the physical environment to determine bias of the robot corresponding to a set of items located in the physical environment while the robot performs the action, the artificial neural network including a biased path of biased nodes having bias weights, a non-biased path of non-biased nodes having non-bias weights, and a limit function, wherein the artificial neural network executes in parallel the biased path of the biased nodes having the bias weights and the non-biased path of the non-biased nodes having the non-bias weights;
identifying, by the computer, equal opportunity and disparate impact on protected attributes from the multimedia data during performance of the action by the robot to weight degree of bias based on a determined change in state of the physical environment in response to the robot performing the action;
performing, by the computer, post processing of the weighted degree of bias to decrease the bias of the robot by merging the biased nodes having the bias weights in the biased path with the non-biased nodes having the non-bias weights in the non-biased path of the artificial neural network and limiting the bias weights using the limit function to form merged nodes having decreased bias;
relabeling, by the computer, training data of a semi-supervised learning model that was used to previously train the robot to perform the action to accomplish the task by the robot in the physical environment based on the post processing of the weighted degree of bias;
retraining, by the computer, the robot to increase performance of the action to accomplish the task by the robot in the physical environment using the relabeled training data;
recalculating, by the computer, a reward corresponding to the action based on the equal opportunity and disparate impact on the protected attributes during performance of the action by the robot; and
updating, by the computer, a Q-table with the recalculated reward corresponding to the action.

2. The computer-implemented method of claim 1 further comprising:
receiving, by the computer, the semi-supervised learning model corresponding to a set of two or more physical environments, wherein the physical environment is one physical environment in the set of two or more physical environments;
training, by the computer, the robot to perform the action to accomplish the task in the physical environment of the set of two or more physical environments based on the semi-supervised learning model; and
mapping, by the computer, the action to be performed by the robot in the physical environment to the reward using the Q-table.

3. The computer-implemented method of claim 2 further comprising:
determining, by the computer, change in state of the physical environment based on the analysis of the multimedia data capturing the robot performing the action to accomplish the task in the physical environment.

4. The computer-implemented method of claim 1 further comprising:
receiving, by the computer, the semi-supervised learning model corresponding to a set of physical environments; and
training, by the computer, a swarm of robots to perform the action to accomplish the task in one or more other physical environments of the set of physical environments based on the semi-supervised learning model.

5. The computer-implemented method of claim 4 further comprising:
receiving, by the computer, via the network, multimedia data capturing the swarm of robots performing the action to accomplish the task in the one or more other physical environments from a second set of sensors;
analyzing, by the computer, the multimedia data capturing the swarm of robots performing the action to accomplish the task in the one or more other physical environments using the artificial neural network; and
determining, by the computer, change in state of the one or more other physical environments based on analysis of the swarm of robots performing the action to accomplish the task in the one or more other physical environments.

6. The computer-implemented method of claim 4 further comprising:
identifying, by the computer, the equal opportunity and disparate impact on protected attributes during performance of the action by the swarm of robots to weight degree of bias based on determined change in state of the one or more other physical environments in response to the performance of the action.

7. The computer-implemented method of claim 1 , wherein the artificial neural network is a convolutional neural network.

8. A computer system for providing fair deep reinforcement learning, the computer system comprising:
a bus system;
a storage device connected to the bus system, wherein the storage device stores program instructions; and
a processor connected to the bus system, wherein the processor executes the program instructions to:
receive, via a network, multimedia data capturing a robot performing an action to accomplish a task in a physical environment from a first set of sensors located in the physical environment;
perform, using an artificial neural network, an analysis of the multimedia data capturing the robot performing the action to accomplish the task in the physical environment to determine bias of the robot corresponding to a set of items located in the physical environment while the robot performs the action, the artificial neural network including a biased path of biased nodes having bias weights, a non-biased path of non-biased nodes having non-bias weights, and a limit function, wherein the artificial neural network executes in parallel the biased path of the biased nodes having the bias weights and the non-biased path of the non-biased nodes having the non-bias weights;
identify equal opportunity and disparate impact on protected attributes from the multimedia data during performance of the action by the robot to weight degree of bias based on a determined change in state of the physical environment in response to the robot performing the action;
perform post processing of a weighted degree of bias to decrease the bias of the robot by merging the biased nodes having the bias weights in the biased path with the non-biased nodes having the non-bias weights in the non-biased path of the artificial neural network and limiting the bias weights using the limit function to form merged nodes having decreased bias;
relabel training data of a semi-supervised learning model that was used to previously train the robot to perform the action to accomplish the task by the robot in the physical environment based on the post processing of the weighted degree of bias;
retrain the robot to increase performance of the action to accomplish the task by the robot in the physical environment using the relabeled training data;
recalculate a reward corresponding to the action based on the equal opportunity and disparate impact on the protected attributes during performance of the action by the robot; and
update a Q-table with the recalculated reward corresponding to the action.

9. The computer system of claim 8 , wherein the processor further executes the program instructions to: receive the semi-supervised learning model corresponding to a set of two or more physical environments, wherein the physical environme

Assignees

• IBM

Inventors

• Baughman Aaron K
• Hammer Stephen C
• Cannon Gray
• Kwatra Shikhar

Classifications

• G06N3/045

  Combinations of networks · CPC title

• G06N3/0895

  Weakly supervised learning, e.g. semi-supervised or self-supervised learning · CPC title

• G06N3/092

  Reinforcement learning · CPC title

• G06N3/0464

  Convolutional networks [CNN, ConvNet] · CPC title

• G06N3/063

  using electronic means · CPC title