Reinforcement learning for h2s abatement

Patent metadata

Field	Value
Publication number	US-2021284551-A1
Application number	US-202117186849-A
Country	US
Kind code	A1
Filing date	Feb 26, 2021
Priority date	Feb 27, 2020
Publication date	Sep 16, 2021
Grant date	—

Abstract

A computer implemented method and a system abates the presence of sulphide (H2S(g), H2S(aq) or HS-(aq)) in a wastewater flowing in a specific wastewater network from an upstream pumping pit to a downstream pumping pit or manhole. The computer implemented method and a system includes dosing into the wastewater at a position upstream of the downstream pit or manhole a chemical for abatement of sulphide, determining by use of a sensor the concentration of sulphide at a position downstream of the position at which chemical is dosed into the wastewater, such as located in the downstream manhole. The amount of chemical dosed is determined by use of a general agent and a specific agent.

First claim

1 . A method for abating the presence of sulphide (H2S(g), H2S(aq) or HS-(aq)) in a wastewater flowing in a specific wastewater network ( 1 ) from an upstream pumping pit ( 9 ) to a downstream pumping pit or manhole ( 4 ) wherein the specific wastewater network being an existing or planned physical realization of a wastewater network, the method comprising
providing the specific wastewater network comprising the upstream pit ( 9 ) and the downstream pit or manhole ( 4 ); providing a dosing unit ( 6 ) configured for dosing a chemical capable of sulphide abatement upstream of the downstream pit or manhole; providing a sensor ( 5 ) downstream of the dosing unit ( 6 ), said sensor being capable of measure concentration of sulphide and provide a readout indicative of the sulphide concentration in a region surrounding the sensor ( 5 ); providing a control unit ( 7 ) configured for providing the dosing unit ( 6 ) with a dosing signal and for receiving as input the readout from the sensor ( 5 ); dosing into the wastewater at a position upstream of the downstream pit or manhole ( 4 ) said chemical for abatement of sulphide, and determining by use of a sensor ( 5 ) the concentration of sulphide at a position downstream of the position at which chemical is dosed into the wastewater, such as located in the downstream manhole ( 4 ); wherein the amount dosed of said chemical for abatement of sulphide is determined by providing a general agent and
training the general agent by reinforcement learning (A), to determine a dosing amount of chemical on the basis of a concentration of sulphide, the training of the general agent (A) is based on numerical simulations of a plurality of real wastewater networks, and/or:
training the/a general agent (A) by reinforcement learning (B), on a numerical simulation of a/said specific wastewater network to determine a dosing amount of chemical on the basis of a concentration of sulphide;
deploying the general agent (A or B) to determine amounts of said chemical for abatement of sulphide to be dosed into said specific wastewater network ( 1 ) and dosing the determined amounts of chemical into the specific wastewater network ( 1 ), wherein the deployment and dosing comprising
training by reinforcement learning the general agent (A, B), to obtain a specific trained agent (C), wherein the reinforcement learning comprising:
initially, determining by use of the general agent (A, B) an amount of said chemical for abatement of sulphide to be dosed on the basis of a determined concentration of sulphide in the specific wastewater network ( 1 ), dosing the determined amount into the wastewater, and after dosing of said chemical determining the concentration of sulphide in the specific wastewater system and further train the general agent (A, B) to obtain the specific trained agent (C);

subsequently determining by use of the specific trained agent (C) an amount of said chemical for abatement of sulphide to be dosed on the basis of a determined concentration of sulphide in the specific wastewater network ( 1 ), dosing the determined amount into the wastewater, and after dosing said chemical determining the concentration of sulphide in the specific wastewater system and further train the specific trained agent (C).

2 . A method according to claim 1 , wherein the further training the specific agent comprising an exploratory element, where the specific agent executes a different amount of dosing of chemical to abate sulphide than what would have been determined by the specific agent, preferably the proportion of times to choose an exploratory action is determined by a probability of taking an exploratory step that, preferably starts high during initial learning and is then annealed towards a very low value after some fixed time or when some conditions are met.

3 . A method according to claim 1 , wherein training the general agent comprising an exploratory element, where the general agent executes a different amount of dosing of chemical to abate sulphide than what would have been determined by the general agent, preferably the proportion of times to choose an exploratory action is determined by a probability of taking an exploratory step that, preferably starts high during initial learning and is then annealed towards a very low value after some fixed time or when some conditions are met.

4 . A method according to claim 1 , wherein the reinforcement learning comprises a reinforcement learning reward, preferably based on the negative absolute difference between a pre-selected level of sulphide concentration and an actual determination of sulphide concentration or based on the negative sum of estimated cost of sulphide and the estimated cost of chemical.

5 . A method according to claim 1 , wherein the reinforcement learning is implemented as a reinforcement learning reward routine based on the negative absolute difference between a pre-selected level of sulphide concentration and an actual determination of sulphide concentration.

6 . A method according to claim 1 , wherein the numerical simulation of a wastewater network system(s) is/are based on the basis of a plurality of data sets from real dosing scenario(s) from wastewater network system(s).

7 . A method according to claim 1 , wherein the determined concentration of sulphide is a value timely averaged over a preselected time, such as over 5.0 minutes, such as over 10.0 minutes.

8 . A method according to claim 1 , wherein the determined concentration of sulphide is determined at preselected points in time, such as at regular intervals.

9 . A method according to claim 1 , wherein the general agent (B) is trained on the basis of simulations on at least some characteristics of the specific wastewater network ( 1 ), wherein the characteristics includes one or more of geometries of the networks system ( 1 ), expected timewise load exposure, expected quality, expected rain, specific information, such as houses connected, number of dimension, and policy from agents acting in other wastewater networks system(s).

10 . A method according to claim 1 , wherein the general agent (A, B) and the specific agent (C) use a policy, for determining the best action given the state of the system and its surroundings, trained on Q learning, deep Q learning, model-based algorithms, actor-critique algorithm, federated learning or transfer_learning the state of the system being e.g. the sulphide concentration history and flow history of the system, the time of the week, the rain in the area, the temperature of the waste water.

11 . A method according to claim 1 , wherein the pre-selected level of sulphide concentration is a concentration interval, such as 5.0±0.1 ppm, preferably 4.0±0.1 ppm, such as 3.0±0.1 ppm, and where in the reinforcement learning comprising providing the specific agent (C) with a negative reward if a determined concentration of sulphide is outside the concentration interval.

12 . A method according to claim 1 , wherein the pre-selected level of sulphide concentration is a concentration value, such as 5.0 ppm, preferably 4.0 ppm, such as 3.0 ppm, and where in the reinforcement learning comprising providing the specific agent (C) with a negative reward if a determined concentration of sulphide is larger or smaller than the concentration value.

13 . A method according to claim 1 , wherein the chemical dosed for abatement of sulphide is iron in one of its common oxidation states, such as Fe 2+ , Fe 3+ and/or Fe 6+ , or NO 3 .

14 . A method according to claim 1 , wherein the specific wastewater network further comprising
an inlet ( 14 ) for receiving wastewater provided in the

Assignees

• Grundfos Holding As

Inventors

• Laegdsmand Mette
• Thomsen Nicolai Bæk
• Ryming Alexander Engels

Classifications

• G06N3/044

  Recurrent networks, e.g. Hopfield networks · CPC title

• G06N3/092

  Reinforcement learning · CPC title

• G06N3/096

  Transfer learning · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/098

  Distributed learning, e.g. federated learning · CPC title