Unified control of diverse actions in a wastewater treatment activated sludge system using reinforcement learning for multi-objective optimization

Abstract

The operation of modern wastewater treatment facilities is a balancing act in which a multitude of variables are controlled to achieve a wide range of objectives, many of which are conflicting. This is especially true within secondary activated sludge systems, where significant research and industry effort has been devoted to advance control optimization strategies, both domain-driven and data-driven. Among data-driven control strategies, reinforcement learning (RL) stands out for its ability to achieve better than human performance in complex environments. While RL has been applied to activated sludge process optimization in existing literature, these applications are typically limited in scope, and never for the control of more than three actions. Expanding the scope of RL control has the potential to increase the optimization potential while concurrently reducing the number of control systems that must be tuned and maintained by operations staff. This study examined several facets of the implementation of multi-action, multi-objective RL agents, namely how many actions a single agent could successfully control and what extent of environment data was necessary to train such agents. This study observed improved control optimization with increasing action scope, though control of waste activated sludge remains a challenge. Furthermore, agents were able to maintain a high level of performance under decreased observation scope, up to a point. When compared to baseline control of the Benchmark Simulation Model No. 1 (BSM1), an RL agent controlling seven individual actions improved the average BSM1 performance metric by 8.3 %, equivalent to an annual cost savings of $40,200 after accounting for the cost of additional sensors.