Reinforcement learning for H2S abatement

Patent metadata

Field	Value
Publication number	US-11772985-B2
Application number	US-202117186849-A
Country	US
Kind code	B2
Filing date	Feb 26, 2021
Priority date	Feb 27, 2020
Publication date	Oct 3, 2023
Grant date	Oct 3, 2023

Abstract

A computer implemented method and a system abates the presence of sulphide (H2S(g), H2S(aq) or HS-(aq)) in a wastewater flowing in a specific wastewater network from an upstream pumping pit to a downstream pumping pit or manhole. The computer implemented method and a system includes dosing into the wastewater at a position upstream of the downstream pit or manhole a chemical for abatement of sulphide, determining by use of a sensor the concentration of sulphide at a position downstream of the position at which chemical is dosed into the wastewater, such as located in the downstream manhole. The amount of chemical dosed is determined by use of a general agent and a specific agent.

First claim

The invention claimed is:

1. A computer-implemented method for abating the presence of sulphide (H2S(g), H2S(aq) or HS-(aq)) in a wastewater flowing in a specific wastewater network from an upstream pumping pit to a downstream pumping pit or manhole wherein the specific wastewater network being an existing or planned physical realization of a wastewater network, the method comprising:
providing the specific wastewater network comprising the upstream pit and the downstream pit or manhole;
providing a dosing unit configured for dosing a chemical capable of sulphide abatement upstream of the downstream pit or manhole;
providing a sensor downstream of the dosing unit, said sensor being capable of measure concentration of sulphide and provide a readout indicative of the sulphide concentration in a region surrounding the sensor;
providing a control unit configured for providing the dosing unit with a dosing signal and for receiving as input the readout from the sensor;
dosing into the wastewater at a position upstream of the downstream pit or manhole said chemical for abatement of sulphide; and
determining by use of the sensor the concentration of sulphide at a position downstream of the position at which chemical is dosed into the wastewater;

wherein the amount dosed of said chemical for abatement of sulphide is determined by
providing a general agent being a software implement algorithm and at least one of:
training the general agent by reinforcement learning (A), to determine a dosing amount of chemical for abatement of H 2 S on the basis of a concentration of sulphide, the training of the general agent (A) is based on numerical simulations of a plurality of real wastewater networks;
training the general agent (A) by reinforcement learning (B), on a numerical simulation of the specific wastewater network to determine a dosing amount of chemical for abatement of H 2 S on the basis of a concentration of sulphide;
training the general agent by reinforcement learning (B), on a numerical simulation of another specific wastewater network to determine a dosing amount of chemical for abatement of H 2 S on the basis of a concentration of sulphide;

deploying the general agent (A or B) to determine amounts of said chemical for abatement of sulphide to be dosed into said specific wastewater network and dosing the determined amounts of chemical into the specific wastewater network by the dosing unit, wherein the deployment and dosing comprises:
training by reinforcement learning the general agent (A, B), to obtain a specific trained agent (C), the specific trained agent (C) being a computer implemented algorithm, wherein the reinforcement learning comprises:
initially, determining by use of the general agent (A, B) an amount of said chemical for abatement of sulphide to be dosed by the dosing unit on the basis of a determined concentration of sulphide in the specific wastewater network by the sensor, dosing the determined amount into the wastewater by the dosing unit, and after dosing of said chemical determining the concentration of sulphide in the specific wastewater system by the sensor and further train the general agent (A, B) to obtain the specific trained agent (C);

subsequently determining by use of the specific trained agent (C) an amount of said chemical for abatement of sulphide to be dosed on the basis of a determined concentration of sulphide in the specific wastewater network, dosing the determined amount into the wastewater by the dosing unit, and after dosing said chemical determining the concentration of sulphide in the specific wastewater system by the sensor and further train the specific trained agent (C).

2. A method according to claim 1 , wherein the further training the specific agent comprising an exploratory element, where the specific agent comprising the exploratory element executes a different amount of dosing of chemical to abate sulphide than what would have been determined by the specific agent without the exploratory element.

3. A method according to claim 1 , wherein training the general agent comprising an exploratory element, where the general agent comprising the exploratory element executes a different amount of dosing of chemical to abate sulphide than what would have been determined by the general agent without the exploratory element.

4. A method according to claim 1 , wherein the reinforcement learning comprises a reinforcement learning reward.

5. A method according to claim 1 , wherein the reinforcement learning is implemented as a reinforcement learning reward routine based on the negative absolute difference between a pre-selected level of sulphide concentration and an actual determination of sulphide concentration.

6. A method according to claim 1 , wherein the numerical simulation of a wastewater network system(s) is/are based on the basis of a plurality of data sets from real dosing scenario(s) from wastewater network system(s).

7. A method according to claim 1 , wherein the determined concentration of sulphide is a value timely averaged over a preselected time.

8. A method according to claim 1 , wherein the determined concentration of sulphide is determined at preselected points in time.

9. A method according to claim 1 , wherein the general agent (B) is trained on the basis of simulations on at least some characteristics of the specific wastewater network, wherein the characteristics includes one or more of geometries of the specific network system, expected timewise load exposure, expected quality, expected rain, specific information including houses connected, number of dimension, and policy from agents acting in other wastewater network system(s).

10. A method according to claim 1 , wherein the general agent (A, B) and the specific agent (C) use a policy, for determining the best action given the state of the system and its surroundings, trained on Q learning, deep Q learning, model-based algorithms, actor-critique algorithm, federated learning or transfer_learning the state of the system being e.g. the sulphide concentration history and flow history of the system, the time of the week, the rain in the area, the temperature of the waste water.

11. A method according to claim 1 , wherein the pre-selected level of sulphide concentration is a concentration interval, and where in the reinforcement learning comprising providing the specific agent (C) with a negative reward if a determined concentration of sulphide is outside the concentration interval.

12. A method according to claim 1 , wherein the pre-selected level of sulphide concentration is a concentration value, and where in the reinforcement learning comprising providing the specific agent (C) with a negative reward if a determined concentration of sulphide is larger or smaller than the concentration value.

13. A method according to claim 1 , wherein the chemical dosed for abatement of sulphide is iron in one of its common oxidation states.

14. A method according to claim 1 , wherein the specific wastewater network further comprising:
an inlet for receiving wastewater provided in the upstream pumping pit;
an outlet for discharging wastewater provided in the downstream pumping pit or manhole;
a pumping main fluidicly connecting the upstream pumping pit with the downstream pumping pit or manhole;
a pump arranged to pump wastewater from the upstream pumping pit to the downstream pumping pit or manhole through the pumping main, said pump being configured to pump in response to receiving a pump control signal,

and wherein:
said sensor is arranged downstream of the position at which chemical is dosed into the wastewater,

Assignees

• Grundfos Holding As

Inventors

• Laegdsmand Mette
• Thomsen Nicolai Bæk
• Ryming Alexander Engels

Classifications

• G06N3/092

  Reinforcement learning · CPC title

• G06N3/096

  Transfer learning · CPC title

• G06N3/0442

  characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] · CPC title

• G06N3/098

  Distributed learning, e.g. federated learning · CPC title

• C02F1/008 Primary

  Control or steering systems not provided for elsewhere in subclass C02F · CPC title