Attack detection and countermeasure identification system

Patent metadata

Field	Value
Publication number	US-12524530-B2
Application number	US-202218089373-A
Country	US
Kind code	B2
Filing date	Dec 27, 2022
Priority date	Dec 27, 2021
Publication date	Jan 13, 2026
Grant date	Jan 13, 2026

Abstract

A method is disclosed which comprises accessing a detector model that is trained in parallel with an operator model and an attacker model using a reinforcement learning technique based on iteratively simulating scenarios of operation of an environment to generate training data and learning weights of the models based on the simulated training data. The simulating of a scenario is based on the last learned weights of the models. The method further comprises, during operation of the environment, applying the detector model to an operator action, a prior observation of state of the environment from prior to taking the operator action, and a current observation of the environment from after taking the operator action, to detect whether an attack on the environment has occurred.

First claim

What is claimed is:

1 . A method performed by one or more computing systems to support responding to a cyberattack on a physical infrastructure system via a computer network environment, the method comprising:
accessing a specification of the physical infrastructure system that includes components having a plurality of states; running scenarios, comprising virtual simulations of the physical infrastructure system that output machine learning model training data, to modify a current state of the physical infrastructure system corresponding to the plurality of states, wherein running a scenario includes:
modifying the current state of the physical infrastructure system based on an operator action, wherein a modification to the plurality of states includes a simulated change to a physical infrastructure topology;
modifying the modified current state of the physical infrastructure system based on an attacker action to generate a new state; and
detecting within the scenario whether an attack on the physical infrastructure system has occurred based on the operator action, the current state, and the new state;
and
training an operator model and a detector model based on the operator action, the attacker action, and a detection of whether an attack on the physical infrastructure system has occurred, wherein the operator model is trained to identify an effective operator action given a particular current state of the physical infrastructure system and the detector model is trained to detect an attack on the physical infrastructure system and said training modifies weights assigned to the operation action and the attacker action as associated with the particular current state and the detection of the attack respectively.

2 . The method of claim 1 , further comprising:
training an attacker model in parallel with training the operator model and the detector model, based on the operator actions, the attacker actions, and the detections of the scenarios, wherein the attacker model is trained to identify effective attacks on the physical infrastructure system.

3 . The method of claim 1 , wherein the running a scenario generates an operator reward for each operator action as an indication of effectiveness of the operator action, an attacker reward for each attacker action as an indication of effectiveness of the attacker action, and a detector reward as an indication of effectiveness of the detection, and wherein the training the operator model and the detector model step factors in the operator reward, the attacker reward, and the detector reward.

4 . The method of claim 1 , further comprising:
receiving a current state of a non-simulated, real environment of a physical infrastructure system, an operator action to modify the current state, and a new state after modification of the current state; and applying the detector model to the operator action, the current state, and the new state to detect whether an attack has occurred on the non-simulated, real environment of the physical infrastructure system.

5 . The method of claim 4 , further comprising:
applying the operator model to identify an effective operator action when an attack is detected.

6 . The method of claim 1 , wherein the running scenarios and the training are performed iteratively, wherein the running employs the operator model, an attacker model, and the detector model that was last trained, respectively, to generate operator actions, to generate an attacker action, and to detect an attack.

7 . The method of claim 1 , wherein the physical infrastructure system is a power grid system includes generators, loads, substations, and lines.

8 . The method of claim 1 , wherein the computer network environment is an information technology (IT) environment.

9 . A processing system configured to support responding to a cyberattack on a physical infrastructure system via a computer network environment, the system comprising:
at least one processor; and at least one non-transitory computer-readable storage medium storing instructions, execution of which by the at least one processor causes the processing system to perform operations comprising:
accessing a specification of the physical infrastructure system that includes components having a plurality of states;
running scenarios, comprising virtual simulations of the physical infrastructure system that output machine learning model training data, to modify a current state of the physical infrastructure system corresponding to the plurality of states, wherein running a scenario includes:
modifying the current state of the physical infrastructure system based on an operator action, wherein a modification to the plurality of states includes a simulated change to a physical infrastructure topology;
modifying the modified current state of the physical infrastructure system based on an attacker action to generate a new state; and
detecting within the scenario whether an attack on the physical infrastructure system has occurred based on the operator action, the current state, and the new state; and
training an operator model and a detector model based on the operator action, the attacker action, and a detection of whether an attack on the physical infrastructure system has occurred, wherein the operator model is trained to identify an effective operator action given a particular current state of the physical infrastructure system and the detector model is trained to detect an attack on the physical infrastructure system and said training modifies weights assigned to the operation action and the attacker action as associated with the particular current state and the detection of the attack respectively.

10 . The system of claim 9 , the operations further comprising:
training an attacker model in parallel with training the operator model and the detector model, based on the operator actions, the attacker actions, and the detections of the scenarios, wherein the attacker model is trained to identify effective attacks on the physical infrastructure system.

11 . The system of claim 9 , wherein the running a scenario generates an operator reward for each operator action as an indication of effectiveness of the operator action, an attacker reward for each attacker action as an indication of effectiveness of the attacker action, and a detector reward as an indication of effectiveness of the detection, and wherein the training the operator model and the detector model step factors in the operator reward, the attacker reward, and the detector reward.

12 . The system of claim 9 , the operations further comprising:
receiving a current state of a non-simulated, real environment of a physical infrastructure system, an operator action to modify the current state, and a new state after modification of the current state; and applying the detector model to the operator action, the current state, and the new state to detect whether an attack has occurred on the non-simulated, real environment of the physical infrastructure system.

13 . The system of claim 12 , the operations further comprising:
applying the operator model to identify an effective operator action when an attack is detected.

14 . The system of claim 9 , wherein the running scenarios and the training are performed iteratively, wherein the running employs the operator model, an attacker model, and the detector model that was last trained, respectively, to generate operator actions, to generate an attacker action, and to detect an attack.

15 . The system of claim 9 , wherein the physical infrastructure system is a power grid system and th

Assignees

• L Livermore Nat Security Llc
• Lawrence Livermore Nat Security

Inventors

• Watson Jean-Paul
• Fox Alyson Lindsey
• Mackay Sarah Camille Mousley
• Mitchell Wayne Bradford
• Landen Matthew
• Chung Key-Whan
• Reed Elizabeth Diane

Classifications

• G06N3/092

  Reinforcement learning · CPC title

• G06F2221/034

  Test or assess a computer or a system · CPC title

• G06N20/00

  Machine learning · CPC title

• G06N7/01

  Probabilistic graphical models, e.g. probabilistic networks · CPC title

• G06F21/554 Primary

  involving event detection and direct action · CPC title