A Review of Research on Coordinated Control of Traffic Signals at Urban Road Intersection Groups

Managing mixed traffic at signalized intersections: An adaptive signal control and CAV coordination system based on deep reinforcement learning

Study on deep reinforcement learning techniques for building energy consumption forecasting

Decentralized network level adaptive signal control by multi-agent deep reinforcement learning

Deploying mobile application tasks that require a lot of computing and are time-sensitive to distant cloud-based data centers has become a popular method of working around the limitations of mobile devices (MDs). Deep reinforcement learning (DRL) techniques for offloading in mobile edge computing (MEC) environments struggle to adapt to new situations due to low sample efficiency for each new context. To address these issues, a novel computational offloading in mobile edge computing (COOL-MEC) algorithm has been proposed that combines the benefits of attention modules and bi-directional long short-term memory. This algorithm improves server resource utilization by lowering the cost of assimilating processing latency, processing energy consumption, and task throughput of latency-sensitive tasks. The experiment's findings show that, when used as intended, the recommended COOL-MEC algorithm minimizes energy consumption. When compared to the current deep convolutional attention reinforcement learning with adaptive reward policy (DCARL-ARP) and DRL techniques, the energy consumption of the proposed COOL-MEC is decreased by 0.06% and 0.08%, respectively. The average time per channel utilized for the execution of the proposed COOL-MEC also decreased by 0.051% and 0.054% when compared with existing DCARL-ARP and DRL methods respectively.

Enhancing traffic signal control with composite deep intelligence

International audience

This review paper explores the significance of machine learning (ML), deep learning (DL), reinforcement learning (RL), and deep reinforcement learning (DRL) techniques in improving traffic management based on cloud and mobile edge computing (MEC) architectures. The key findings and contributions of this review highlight the potential of these techniques for transforming traffic management systems through data-driven decision-making, adaptive control, and optimization. The challenges identified in this field include data availability and quality, scalability and computational requirements, privacy and security concerns, and ethical considerations. In conclusion, ML, DL, RL, and DRL techniques, in conjunction with cloud and MEC architectures, have significant implications for improving traffic management. Their ability to process and analyse large-scale and real-time traffic data enables improved traffic flow, reduced congestion, enhanced energy efficiency, and enhanced overall transportation system performance.

Deep Reinforcement Learning (DRL) is a rapidly evolving field that has significantly influenced autonomous systems such as self-driving cars, drones, and robotics. This survey aims to provide a comprehensive overview of the state-of-the-art DRL techniques and their applications in real-world autonomous systems. The paper discusses various architectures such as Deep Q-Networks (DQNs), Policy Gradient methods, and Actor-Critic models, analyzing their strengths and weaknesses in dynamic and complex environments. Moreover, the study focuses on how DRL can address specific challenges in decision-making, navigation, and obstacle avoidance for autonomous vehicles. The integration of DRL with sensor data, such as LIDAR and camera inputs, is explored to understand how these systems can learn more efficiently in real-time environments. Furthermore, the paper examines the scalability of DRL models in large-scale autonomous systems and presents the most recent advancements in this domain. Challenges such as overfitting, reward shaping, and sample inefficiency are discussed, alongside potential future directions like multi-agent systems and cooperative DRL. The review also highlights real-world applications and case studies, illustrating how DRL is implemented in autonomous systems. Finally, the ethical and safety concerns associated with autonomous DRL systems are considered, particularly in the context of self-driving cars and other autonomous technologies that interact with humans.

Cyber-physical systems (CPS) play a vital role in modern society across various sectors, ranging from smart grid to water treatment, and their security has become one of the major concerns. Due to the significantly growing complexity and scale of CPS and cyber-attacks, it is imperative to develop defense and prevention strategies specifically for CPS that are adaptive, scalable, and robust. An important research and application direction in this domain is time series anomaly detection within CPS utilizing advanced machine learning techniques, such as deep learning and reinforcement learning. However, many anomaly detectors fail to balance between detection performance and computational overhead, limiting their applicability in CPS. In this paper, we introduce a novel agent-based dynamic thresholding (ADT) method based on the deep reinforcement learning technique, i.e. deep Q-network (DQN), to model thresholding in anomaly detection as a Markov decision process. By utilizing anomaly scores generated from an autoencoder and other useful information perceived from a simulated environment, ADT performs the optimal dynamic thresholding control, facilitating real-time adaptive anomaly detection for time series. Rigorous evaluations were conducted on realistic datasets from water treatment and industrial control systems, specifically SWaT, WADI, and HAI, comparing against established benchmarks. The experimental results demonstrate ADT's superior detection performance, dynamic thresholding capability, data-efficient learning, and robustness. Notably, ADT, even when trained on minimal labeled data, consistently outperforms benchmarks with F1 scores ranging from 0.995 to 0.999 across all datasets. It is effective even in challenging scenarios where the environmental feedback is noisy, delayed, or partial. Beyond its direct application as an advanced anomaly detector, ADT possesses the versatility to act as a lightweight dynamic thresholding controller, boosting other anomaly detection models. This underscores ADT's considerable promise in sophisticated and dynamic CPS environments.

The growing IoT landscape requires effective server deployment strategies to meet demands including real-time processing and energy efficiency. This is complicated by heterogeneous, dynamic applications and servers. To address these challenges, we propose ReinFog, a modular distributed software empowered with Deep Reinforcement Learning (DRL) for adaptive resource management across edge/fog and cloud environments. ReinFog enables the practical development/deployment of various centralized and distributed DRL techniques for resource management in edge/fog and cloud computing environments. It also supports integrating native and library-based DRL techniques for diverse IoT application scheduling objectives. Additionally, ReinFog allows for customizing deployment configurations for different DRL techniques, including the number and placement of DRL Learners and DRL Workers in large-scale distributed systems. Besides, we propose a novel Memetic Algorithm for DRL Component (e.g., DRL Learners and DRL Workers) Placement in ReinFog named MADCP, which combines the strengths of Genetic Algorithm, Firefly Algorithm, and Particle Swarm Optimization. Experiments reveal that the DRL mechanisms developed within ReinFog have significantly enhanced both centralized and distributed DRL techniques implementation. These advancements have resulted in notable improvements in IoT application performance, reducing response time by 45%, energy consumption by 39%, and weighted cost by 37%, while maintaining minimal scheduling overhead. Additionally, ReinFog exhibits remarkable scalability, with a rise in DRL Workers from 1 to 30 causing only a 0.3-second increase in startup time and around 2 MB more RAM per Worker. The proposed MADCP for DRL component placement further accelerates the convergence rate of DRL techniques by up to 38%.

Control methodologies of traffic signals have significantly improved during the recent past along with advancements in technology. Adaptive traffic signal control is the most recent and advanced control type of traffic signals. Adaptive control is able to efficiently relieve traffic congestion by continuously adjusting signal timings according to real-time traffic conditions. Conventional optimization methods such as integer programming, hill climbing, or descent gradient searching have been gradually overshadowed by genetic algorithms in many areas including traffic signal operation. The research presented in this article is distinct from previous studies in that it focuses on real-time adaptive signal optimization using genetic algorithms. The proposed adaptive signal system provides acyclic signal operation based on a rolling horizon real-time control approach. The algorithm was tested using microsimulation for on-line evaluation and comparison to fixed-time plans generated from the latest TRANSYT-7F version 9.7, which has a genetic optimization feature. The developed signal system consists of three major components including a genetic algorithm optimization module, an internal traffic simulation module, and a database management system all working in cooperation to optimize signal timings in real time. Using the pseudo on-line simulation platform, three testing scenarios for high, medium, and low level of traffic demands were conducted focusing on evaluating several important features of the proposed adaptive signal control system. The test results indicated that real-time genetic control outperformed fixed-signal timing plan in all scenarios based on total vehicle delay.

In the traffic signal light control method based on deep reinforcement learning, there is a challenging problem to coordinate control traffic signal because of the high complexity of the road network. This paper proposes a minimize pressure difference traffic signal control based on deep reinforcement learning. It use traffic pressure difference at each phase of the intersection as the reward. The pressure value considers the queue length of vehicles and the capacity of the roads. The traffic state of the intersection is converted into a two-dimensional matrix composed of vehicle position and speed information, the control strategy is learned online through the deep double Q-learning network model to realize the adaptive control of the traffic signal. The algorithm proposed in this paper is simulated by the traffic simulation platform SUMO and compared with two existing traffic signal control methods based on deep reinforcement learning. The experimental results show that the algorithm has a faster convergence rate and better performance in regional traffic signal coordination control.

The prediction of hidden or missing links in a criminal network, which represent possible interactions between individuals, is a significant problem. The criminal network prediction models commonly rely on Social Network Analysis (SNA) metrics. These models leverage on machine learning (ML) techniques to enhance the predictive accuracy of the models and processing speed. The problem with the use of classical ML techniques such as support vector machine (SVM), is the dependency on the availability of large dataset for training purpose. However, recent ground breaking advances in the research of deep reinforcement learning (DRL) techniques have developed methods of training ML models through self-generated dataset. In view of this, DRL could be applied to other domains with relatively smaller dataset such as criminal networks. Prior to this research, few, if any, previous works have explored the prediction of links within criminal networks that could appear and/or disappear over time by leveraging on DRL technique. Therefore, in this paper, the primary objective is to construct a time-based link prediction model (TDRL) by leveraging on DRL technique to train using a relatively small real-world criminal dataset that evolves over time. The experimental results indicate that the predictive accuracy of the DRL model trained on the temporal dataset is significantly better than other ML models that are trained only with the dataset at specific snapshot in time.

The recent breakthroughs of deep reinforcement learning (DRL) technique in Alpha Go and playing Atari have set a good example in handling large state and actions spaces of complicated control problems. The DRL technique is comprised of (i) an offline deep neural network (DNN) construction phase, which derives the correlation between each state-action pair of the system and its value function, and (ii) an online deep Q-learning phase, which adaptively derives the optimal action and updates value estimates. In this paper, we first present the general DRL framework, which can be widely utilized in many applications with different optimization objectives. This is followed by the introduction of three specific applications: the cloud computing resource allocation problem, the residential smart grid task scheduling problem, and building HVAC system optimal control problem. The effectiveness of the DRL technique in these three cyber-physical applications have been validated. Finally, this paper investigates the stochastic computing-based hardware implementations of the DRL framework, which consumes a significant improvement in area efficiency and power consumption compared with binary-based implementation counterparts.

Performance optimization of criminal network hidden link prediction model with deep reinforcement learning