Systems and methods for executing confidence-aware reinforcement learning

Patent metadata

Field	Value
Publication number	US-2025165796-A1
Application number	US-202318516125-A
Country	US
Kind code	A1
Filing date	Nov 21, 2023
Priority date	Nov 21, 2023
Publication date	May 22, 2025
Grant date	—

Abstract

Methods and systems for executing confidence-aware reinforcement learning for an Artificial Intelligence (AI) model for subsequent deployment of that AI model in an environment are disclosed. The method includes accessing a set of expert trajectories, each expert trajectory comprising a sequence of expert state-action pairs, the expert entities complying with an expert constraint that is unknown. The method also includes generating a main constraint for the set of expert trajectories, the main constraint being conditioned on a pre-determined confidence level, the pre-determined confidence level being indicative of a probability that the main constraint is at least as constraining as the expert constraint, the main constraint comprising one or more rules limiting the actions that are executable by the AI model, determining a target policy among a plurality of policies, the target policy complying with the main constraint and executing the target policy by the AI model.

First claim

1 . A computer-implemented method for executing confidence-aware reinforcement learning for an Artificial Intelligence (AI) model for subsequent deployment of that AI model in an environment, the method comprising:
accessing a set of expert trajectories, each expert trajectory comprising a sequence of expert state-action pairs, a given one of the expert trajectories including information about a given state of the environment and a corresponding action that is to be executed in response to the given state, the expert entities complying with an expert constraint that is unknown; generating a main constraint for the set of expert trajectories, the main constraint being conditioned on a pre-determined confidence level, the pre-determined confidence level being indicative of a probability that the main constraint is at least as constraining as the expert constraint, the main constraint comprising one or more rules limiting the actions that are executable by the AI model; determining a target policy among a plurality of policies, the target policy complying with the main constraint; and executing the target policy by the AI model.

2 . The method of claim 1 , further comprising:
accessing a set of policies, each policy being a mapping from states to actions for the sequences of expert state-action pairs of the expert trajectories, an execution of the policy aiming at maximizing a reward; determining a policy complying with the main constraint; executing the policy by iteratively:
executing the actions of the policy,
receiving indication of rewards from and states of the environment, and
adjusting the policy based on outcomes of the actions and received rewards.

3 . The method of claim 2 , further comprising, prior to executing the policy:
determining a policy-value of the target policy; in response to the policy-value being below a pre-determined value threshold, flagging the set of expert trajectories as insufficient.

4 . The method of claim 3 , further comprising augmenting the set of expert trajectories with additional expert trajectories until the policy-value exceeds the pre-determined value threshold.

5 . The method of claim 3 , wherein determining the policy-value of the policy comprises determining an expected cumulative reward based on rewards associated with the action-state pairs of the policy.

6 . The method of claim 1 , wherein generating a main constraint for the set of expert trajectories comprises:
determining a constraint distribution based on the set of expert trajectories; selecting a constraint from the constraint distribution based on the pre-determined confidence level as the main constraint.

7 . The method of claim 6 , wherein selecting the constraint from the constraint distribution comprises selecting the lower boundary constraint of a quantile of the constraint distribution based on the pre-determined confidence level.

8 . The method of claim 7 , wherein the main constraint is:
quantile P(c) (1−λ)

where P(c) is the constraint distribution and λ is the pre-determined confidence level.

9 . The method of claim 6 , wherein determining a constraint distribution comprises:
employing a neural network encoding the set of expert trajectories to determine, for each of the expert trajectory, a set of contribution factors; and adjusting a template distribution according to the set of contribution factors to form the constraint distribution.

10 . The method of claim 9 , wherein each expert trajectory is encoded with a corresponding encoder having corresponding weights in the neural network.

11 . A system for executing confidence-aware reinforcement learning for an Artificial Intelligence (AI) model for subsequent deployment of that AI model in an environment, the system comprising a controller and a memory storing a plurality of executable instructions which, when executed by the controller, cause the system to:
access a set of expert trajectories, each expert trajectory comprising a sequence of expert state-action pairs, a given one of the expert trajectories including information about a given state of the environment and a corresponding action that is to be executed in response to the given state, the expert entities complying with an expert constraint that is unknown; generate a main constraint for the set of expert trajectories, the main constraint being conditioned on a pre-determined confidence level, the pre-determined confidence level being indicative of a probability that the main constraint is at least as constraining as the expert constraint, the main constraint comprising one or more rules limiting the actions that are executable by the AI model; determine a target policy among a plurality of policies, the target policy complying with the main constraint; and execute the target policy by the AI model.

12 . The system of claim 11 , wherein the system is further configured to:
access a set of policies, each policy being a mapping from states to actions for the sequences of expert state-action pairs of the expert trajectories, an execution of the policy aiming at maximizing a reward; determine a policy complying with the main constraint; execute the policy by iteratively:
executing the actions of the policy,
receiving indication of rewards from and states of the environment, and
adjusting the policy based on outcomes of the actions and received rewards.

13 . The system of claim 12 , wherein the system is further configured to, prior to executing the policy:
determine a policy-value of the target policy; in response to the policy-value being below a pre-determined value threshold, flag the set of expert trajectories as insufficient.

14 . The system of claim 13 , wherein the system is further configured to augment the set of expert trajectories with additional expert trajectories until the policy-value exceeds the pre-determined value threshold.

15 . The system of claim 13 , wherein the system is further configured to, upon determining the policy-value of the policy, determine an expected cumulative reward based on rewards associated with the action-state pairs of the policy.

16 . The system of claim 11 , wherein the system is further configured to, upon generating a main constraint for the set of expert trajectories:
determine a constraint distribution based on the set of expert trajectories; select a constraint from the constraint distribution based on the pre-determined confidence level as the main constraint.

17 . The system of claim 16 , wherein the system is further configured to select the constraint from the constraint distribution by selecting the lower boundary constraint of a quantile of the constraint distribution based on the pre-determined confidence level.

18 . The system of claim 17 , wherein the main constraint is:
quantile P(c) (1−λ)

where P(c) is the constraint distribution and λ is the pre-determined confidence level.

19 . The system of claim 16 , wherein the system is further configured to, upon determining a constraint distribution:
employ a neural network encoding the set of expert trajectories to determine, for each of the expert trajectory, a set of contribution factors; and adjust a template distribution according to the set of contribution factors to form the constraint distribution.

20 . The system of claim 19 , wherein each expert trajectory is encoded with a corresponding encoder having corresponding weights in the neural network.

Assignees

• Liu Guiliang
• Poupart Pascal
• Huawei Tech Canada Co Ltd

Inventors

• Poupart Pascal
• Liu Guiliang
• Elmahgiubi Mohammed

Classifications

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06N3/045

  Combinations of networks · CPC title

• G06N3/08

  Learning methods · CPC title

• G06N3/006

  based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO] · CPC title

• G06N3/092 Primary

  Reinforcement learning · CPC title