Dynamic User Equilibrium Research Articles

Multi-agent reinforcement learning for Markov routing games: A new modeling paradigm for dynamic traffic assignment

This paper aims to develop a paradigm that models the learning behavior of intelligent agents (including but not limited to autonomous vehicles, connected and automated vehicles, or human-driven vehicles with intelligent navigation systems where human drivers follow the navigation instructions completely) with a utility-optimizing goal and the system’s equilibrating processes in a routing game among atomic selfish agents. Such a paradigm can assist policymakers in devising optimal operational and planning countermeasures under both normal and abnormal circumstances. To this end, we develop a Markov routing game (MRG) in which each agent learns and updates her own en-route path choice policy while interacting with others in transportation networks. To efficiently solve MRG, we formulate it as multi-agent reinforcement learning (MARL) and devise a mean field multi-agent deep Q learning (MF-MA-DQL) approach that captures the competition among agents. The linkage between the classical DUE paradigm and our proposed Markov routing game (MRG) is discussed. We show that the routing behavior of intelligent agents is shown to converge to the classical notion of predictive dynamic user equilibrium (DUE) when traffic environments are simulated using dynamic loading models (DNL). In other words, the MRG depicts DUEs assuming perfect information and deterministic environments propagated by DNL models. Four examples are solved to illustrate the algorithm efficiency and consistency between DUE and the MRG equilibrium, on a simple network without and with spillback, the Ortuzar Willumsen (OW) Network, and a real-world network near Columbia University’s campus in Manhattan of New York City.

Modeling traffic dynamics in periphery-downtown urban networks combining Vickrey's theory with Macroscopic Fundamental Diagram: user equilibrium, system optimum, and cordon pricing

This paper constructs a dynamic model for a periphery-downtown urban network combining Vickrey's theory with Macroscopic Fundamental Diagram (MFD) to capture interactions between the peripheral traffic and the downtown, and to examine how travelers on different corridors compete for the downtown road resources in the context of dynamic user equilibrium. In no-toll equilibrium if existing, when there exists no congestion delay in the downtown area, the departures for each corridor are regulated only by its bottleneck's capacity on the periphery of the downtown area, as occurred in Vickrey's bottleneck model, whereas when the downtown accumulation goes beyond the critical value, the departures for each corridor are constrained not only by its bottleneck's capacity but also by the downtown congestion level, regulated by all corridors’ departures in turn. We show that under specific assumptions, at the system optimum, the entrances at the boundary of the downtown area should be running at their full capacities before the onset of the downtown congestion, and the downtown network should be operating upon the onset of the downtown congestion at the critical accumulation with maximum production and highest traveling speed. Two optimal time-varying cordon pricing schemes (entrance-independent and entrance-dependent respectively) are developed to minimize the total social cost, which includes the queuing time costs concerning all corridors, the moving time cost in the interior of the downtown area, and schedule delay cost. What's more, the two time-varying cordon tolls to support the system optimum are analyzed from practical views. In the end, analytical results are illustrated and verified with numerical experiments.

Simulation-Based Dynamic Traffic Assignment with Continuously Distributed Value of Time for Heterogeneous Users

The value of time (VOT) attribute is usually utilized to represent the trade-off between time and monetary expenses in transportation problems. A good representation of VOT is essential for evaluation of any road pricing scheme. Conventionally, in dynamic traffic assignment models, VOT is considered as either constant or finite discrete among travelers because of memory and computational limitations, which in turn could introduce bias in the results. This research explicitly models the individual bi-criteria dynamic user equilibrium (IBDUE) problem and presents a distinct simulation-based solution algorithm that enables individual-based traffic assignment within reasonable run time with a successful implementation of variable and continuously distributed VOT in a simulation-based dynamic traffic assignment package. Numerical analysis reveals that the constant and discrete VOT models tend to overestimate toll road usage compared with the continuous VOT model when the toll charge is low, and underestimate it when the toll charge is high, which reflects previous studies. In the meantime, an experiment on a real-world congestion pricing scheme demonstrates the capability of the proposed algorithm on evaluating flow-dependent pricing schemes.

Computing Dynamic User Equilibrium on Large-Scale Networks Without Knowing Global Parameters

Dynamic user equilibrium (DUE) is a Nash-like solution concept describing an equilibrium in dynamic traffic systems over a fixed planning period. DUE is a challenging class of equilibrium problems, connecting network loading models and notions of system equilibrium in one concise mathematical framework. Recently, Friesz and Han introduced an integrated framework for DUE computation on large-scale networks, featuring a basic fixed-point algorithm for the effective computation of DUE. In the same work, they present an open-source MATLAB toolbox which allows researchers to test and validate new numerical solvers. This paper builds on this seminal contribution, and extends it in several important ways. At a conceptual level, we provide new strongly convergent algorithms designed to compute a DUE directly in the infinite-dimensional space of path flows. An important feature of our algorithms is that they give provable convergence guarantees without knowledge of global parameters. In fact, the algorithms we propose are adaptive, in the sense that they do not need a priori knowledge of global parameters of the delay operator, and which are provable convergent even for delay operators which are non-monotone. We implement our numerical schemes on standard test instances, and compare them with the numerical solution strategy employed by Friesz and Han.

Convergence of fixed-point algorithms for elastic demand dynamic user equilibrium

In this paper we present sufficient conditions for convergence of projection and fixed-point algorithms used to compute dynamic user equilibrium with elastic travel demand (E-DUE). The assumption of strongly monotone increasing path delay operators is not needed. In its place, we assume path delay operators are merely weakly monotone increasing, a property assured by Lipschitz continuity, while inverse demand functions are strongly monotone decreasing. Lipschitz continuity of path delay is a very mild regularity condition. As such, nonmonotone delay operators may be weakly monotone increasing and satisfy our convergence criteria, provided inverse demand functions are strongly monotone decreasing. We illustrate convergence for nonmonotone path delays via a numerical example.

Charging station planning based on the accumulation prospect theory and dynamic user equilibrium

Large-scale use of electric vehicles will greatly increase the traffic pressure on urban road network. Therefore, planning of charging stations for electric vehicles considering charging demand and transportation network is particularly important for the coordinated development of electric vehicles and intelligent transportation. Under the condition of bounded rationality, this paper considers such factors as the travel utility perception difference between the users of fuel vehicles and electric vehicles, the time-varying of traffic flow, the location and service level of charging stations. On this basis, combining the cumulative prospect theory, dynamic traffic flow allocation and charging demands, a two-level programming model is established to solve the problem of charging station site selection. The upper layer is a system optimal model, the goal is to minimize the travel time of the network. The lower model describes the time-variability of departure time and the randomness of charging and travel behaviors, establishes the dynamic user equilibrium model and designs the heuristic algorithm. The validity of the model and algorithm is verified by a numerical example. Through the simulation experiment, the optimal location scheme of charging station under different electric vehicle proportion is obtained, and the driving characteristics of two types of vehicles are analyzed. Compared with the traditional model, it is found that the charging station planning considering bounded rationality can achieve higher road network traffic efficiency with fewer charging piles.

Link Travel Time Estimation in Double-Queue-Based Traffic Models

Double queue concept has gained its popularity in dynamic user equilibrium (DUE) modeling because it can properly model real traffic dynamics. While directly solving such double-queue-based DUE problems is extremely challenging, an approximation scheme called first-order approximation was proposed to simplify the link travel time estimation of DUE problems in a recent study without evaluating its properties and performance. This paper focuses on directly investigating the First-In-First-Out property and the performance of the first-order approximation in link travel time estimation by designing and modeling dynamic network loading (DNL) on single-line stretch networks. After model formulation, we analyze the First-In-First-Out (FIFO) property of the first-order approximation. Then a series of numerical experiments is conducted to demonstrate the FIFO property of the first-order approximation, and to compare its performance with those using the second-order approximation, a point queue model, and the cumulative inflow and exit flow curves. The numerical results show that the first-order approximation does not guarantee FIFO and also suggest that the second-order approximation is recommended especially when the link exit flow is increasing. The study provides guidance for further study on proposing new methods to better estimate link travel times.

Dynamic user equilibrium for departure time choice in the basic trip-based model

Recent research indicates that the trip-based models can perform more accurately for capturing network hyper-congestion than the conventional macroscopic fundamental diagram (MFD) dynamics, especially during transient phases. However, due to the complex mathematical structure of the trip-based models, deriving analytical properties of the dynamic user equilibrium (DUE) of departure time choice with the trip-based models is still a challenge. This paper investigates the DUE problem of departure time choice in an isotropic urban network with identical travelers. Traffic dynamics is captured by the basic trip-based model using a speed-MFD that maps the traffic accumulation to the space-mean speed of the network. Necessary conditions for dynamic user equilibrium with and without inflow capacity constraint are derived, respectively. Under dynamic user equilibrium condition, no traveler can reduce her/his travel cost by changing the departure time. The analysis reveals the significant difference between the basic trip-based model and the conventional MFD models regarding the information support involved in the departure time choice. The derivation does not rely on some common assumptions in the literature such as linear travel cost function, no late arrivals, or linear speed-MFD. The numerical example indicates that the inflow capacity constraint can help prevent two peaks in the departure profile and the vehicle accumulation.

Comparing Dynamic User Equilibrium and Noniterative Stochastic Route Choice in a Simulation-Based Dynamic Traffic Assignment Model: Practical Considerations for Large-Scale Networks

Simulation-based dynamic traffic assignment (DTA) models play a vital role in transportation planning and operations. While the widely studied equilibrium-seeking DTA including dynamic user equilibrium (DUE) often provides robust and consistent outcomes, their expensive computational cost for large-scale network applications has been a burden in practice. The noniterative stochastic route choice (SRC) model, as a nonequilibrium seeking DTA model, provides an alternative for specific transportation operations applications that may not require equilibrium results after all (e.g., evacuation and major network disruptions) and thus tend to be computationally less expensive, yet may suffer from inconsistent outcomes. While DUE is a widely accepted approach for many strategic planning applications, SRC has been increasingly used in practice for traffic operations purposes. This paper aims to provide a comparative and quantitative analysis of the two modeling approaches. Specifically, a comparison has been made at two levels: link-level flows and network-level congestion patterns. Results suggest that adaptive driving improves the quality of the SRC solution, but its difference from DUE still remains significant at the link level. Results have practical implications for the application of large-scale simulation-based DTA models for planning and operations purposes.

Mixed traffic flow of human driven vehicles and automated vehicles on dynamic transportation networks

Improving the system performance of a traffic network by dynamically controlling the routes of connected and automated vehicles (CAVs) is an appealing profit that CAVs can bring to our society. Considering that there may be a long way to achieve 100% CAV penetration, we discuss in this paper the mixed traffic flow of human driven vehicles (HDVs) and CAVs on a transportation network. We first propose a double queue (DQ) based mixed traffic flow model to describe the link dynamics as well as the flow transitions at junctions. Based on this mixed flow model, we develop a dynamic bi-level framework to capture the behavior and interaction of HDVs and CAVs. This results in an optimal control problem with equilibrium constraints (OCPEC), where HDVs’ route choice behavior is modeled at the lower level by the instantaneous dynamic user equilibrium (IDUE) principle and the CAVs’ route choice is modelled by the dynamic system optimal (DSO) principle at the upper level. We show how to discretize the OCPEC to a mathematical programming with equilibrium constraints (MPEC) and discuss its properties and solution techniques. The non-convex and non-smooth properties of the MPEC make it hard to be efficiently solved. To overcome this disadvantage, we develop a decomposition based heuristic model predictive control (HMPC) method by decomposing the original MPEC problem into two separate problems: one IDUE problem for HDVs and one DSO problem for CAVs. The experiment results show that, compared with the scenario that all vehicles are HDVs, the proposed methods can significantly improve the network performance under the mixed traffic flow of HDVs and CAVs.

Analysis of first-come-first-served mechanisms in one-way car-sharing services

The principle of “first-come-first-served” (FCFS) has been widely adopted in the deployment of car-sharing services (CSS) to manage service requests for the sake of equity. Most studies of CSS do not explicitly model the supply-demand interactions of shared cars, especially when supply insufficiency arises. This study formulates the supply-demand dynamics of one-way CSS under different FCFS mechanisms and embeds them in a boundedly rational dynamic user equilibrium (BR-DUE) problem. Two disaggregate FCFS mechanisms are suggested to improve the utilization of shared cars given the same CSS supplies in the discrete-time domain. To accurately capture the choice of CSS in space and time, a path expansion strategy is proposed to cope with different waiting times under the disaggregate FCFS mechanisms. The path expansion strategy congruently bridges the aggregate-disaggregate analyses and is incorporated in an adaptive column generation algorithm to solve the BR-DUE problem in a bi-modal supernetwork. Numerical examples demonstrate that the FCFS mechanisms have a significant influence on the supply-demand dynamics and choice of CSS.

An agile vehicle‐based dynamic user equilibrium scheme for urban traffic signal control

An agile vehicle‐based dynamic user equilibrium scheme for urban traffic signal control

Incorporating vehicle self-relocations and traveler activity chains in a bi-level model of optimal deployment of shared autonomous vehicles

The combination of autonomous vehicles (AVs) and free-floating car-sharing scheme is expected to deliver high potentials of both through effective AV self-relocations. Little research has been done on the deployment of shared AVs (SAVs) considering the interplays among SAV relocations, supply-demand dynamics, and travelers’ multi-modal multi-activity schedules. This study aims to propose a bi-level system optimal model inclusive of a new hub-based relocation strategy to moderate the supply and demand of SAVs. The lower-level captures travelers’ activity-travel scheduling behavior by an extended dynamic user equilibrium model and the upper-level determines the hub locations, fleet size, and initial distribution of SAVs. A heuristic algorithm based on Lagrangian relaxation is developed to solve the network design problem. Numerical examples demonstrate that SAV relocations can significantly influence travelers’ daily schedules and enhance mobility efficiency in the multi-modal transport system. We also find that the proposed hub-based relocation strategy outperforms two common SAV relocation strategies in the literature.

Minimizing network-wide emissions by optimal routing through inner-city gating

In this paper, we propose a control and optimization framework to reduce network-wide emissions in an urban traffic network. The framework is comprised of two layers. The first layer (optimal green routing) predicts the optimal splitting coefficients for all Origin–Destination (OD) pairs crossing the city center or bypassing. As implementing optimal routing strategies on the field is almost impossible as we do not have direct control over the user’s decisions, we choose to balance travel time through gating at the city perimeter so that the usual Dynamic User Equilibrium (DUE) discipline matches the optimal splitting coefficients. The second layer then adjusts inflows at the city gates to the DUE solution based on the instantaneous travel times corresponding to the optimal routing strategy for each OD pair. The accumulation-based Macroscopic Fundamental Diagram (MFD) model of a single reservoir city with seven arterial routes and six bypass alternatives is developed. A Linear programming problem is formulated to determine the optimal splitting coefficients and a Nonlinear Model Predictive Control (NMPC)-based gating control strategy is designed to track the optimal splitting coefficients. The network-wide emission control framework is compared to three other gating strategies aiming to (i) optimize traffic conditions or (ii) minimize emissions, in the inner-city only, and (iii) optimize traffic conditions in the whole network. A comprehensive analysis conducted on all four approaches is presented. We compare the results of the controlled case with respect to the uncontrolled case and with respect to each other. The comparison results show that: (i) the proposed network-wide emission control strategy significantly outperforms the other two simpler control strategies focusing on inner-city only in reducing the total emission, (ii) but also improves the in total time spent and mean speed at the network-level.

Dynamic equilibrium commuting in a multilane system with ridesharing

High-occupancy vehicle (HOV) lane has been designed to maximize the movement of people rather than vehicles. For completely understanding the travel pattern of a multilane highway system, this paper presents a dynamic user equilibrium framework, in which commuters, composed of solo-driving drivers, ridesharing drivers and ridesharing passengers, choose their departure times and transport modes to workplace every day. The ridesharing vehicles can use both the HOV lane and the general purpose lane while the solo drivers are only allowed to use the latter. It is found that the rush hour on both lanes is identical but the system’s total travel cost decreases, in a nonlinear way, with the expansion of HOV lane’s capacity.

Macroscopic fundamental diagram based perimeter control considering dynamic user equilibrium

Macroscopic fundamental diagram (MFD) has been receiving increasing attention recently due to its potential to describe traffic dynamics and guide the design of traffic control schemes at the network level. Perimeter control and route guidance are two main MFD-based traffic control approaches. However, current MFD-based perimeter control seldom considers travelers’ route choice behavior, while MFD-based route guidance studies usually assume directly that travelers would follow the guidance and neglect the effects of traffic control. This paper aims to integrate the MFD-based perimeter control (i.e., the behavior of a system manager) and the dynamic user equilibrium based route choice behavior (i.e., the behavior of travelers) into one rigorous mathematical framework. Given a traffic network that has been divided into multiple homogeneous regions, we use MFD to describe the dynamics of each region, and use point queue model to capture the dynamics of queues formed at the boundaries. Besides, we model travelers' route choice behavior by the instantaneous dynamic user equilibrium (IDUE) principle, and design an efficient range perimeter control method from the system perspective. We model the interactions between the system manager and the travelers as a non-zero sum, non-cooperative differential game, where the system manager aims to improve the system performance while travelers try to minimize their own travel times. Meanwhile, they share the common constraints (i.e., MFD dynamics and point queue dynamics at boundaries). Mathematically, this leads to a differential complementarity system (DCS). We propose a time-stepping approach to discretize and solve the DCS model, based on which the solution existence and convergence are also established. Numerical results show that the proposed method can limit the vehicle accumulations within the efficient range of each region, which helps improve the network performance. Compared with the condition without perimeter control, the proposed control method can improve network-wide traffic performance up to 14.18%.

ALADIN-Based Coordinated Operation of Power Distribution and Traffic Networks With Electric Vehicles

Electric vehicles (EVs) are recognized as a promising remedy for the environmental crisis and fuel shortage faced by modern metropolises. But meanwhile, with the popularization of EVs, the unordered charging of EVs will have negative impacts on both the power distribution network (DN) and the traffic network (TN). The well-scheduled EV charging/discharging behaviors could participate in the coordinated operation of TN and DN to significantly enhance the energy utilization efficiency of both networks. With an assumption that an entity like state grid capable of dispatching distributed generators and adjusting the charging prices of fast charging stations, a traffic-distribution coordination (TDC) model is proposed to minimize the travel cost of TN and the energy service cost of DN, which simultaneously considers the economic operation of DN by alternating current dynamic optimal power flow and the traffic flow assignment of TN by EVs dynamic user equilibrium, respectively. And afterward, the augmented Lagrangian alternating direction inexact Newton method is adopted to solve the TDC model. Finally, the necessity of the coordinated operation of TN and DN and the effectiveness of the TDC model are validated in an integrated system of modified Nguyen–Dupius TN and IEEE 33-bus DN.

Macroscopic Fundamental Diagram Based Discrete Transportation Network Design

The presence of demand uncertainty brings challenges to network design problems (NDP), because fluctuations in origin-destination (OD) demand have a prominent effect on the corresponding total travel time, which is usually adopted as an index to evaluate the network design problem. Fortunately, the macroscopic fundamental diagram (MFD) has been proved to be a property of the road network itself, independent of the origin-destination demand. Such characteristics of an MFD provide a new theoretical basis to assess the traffic network performance and further appraise the quality of network design strategies. Focusing on improving network capacity under the NDP framework, this paper formulates a bi-level programming model, where at the lower level, flows are assigned to the newly extended network subject to user equilibrium theory, and the upper level determines which links should be added to achieve the maximum network capacity. To solve the proposed model, we design an algorithm framework, where traffic flow distribution of each building strategy is calculated under the dynamic user equilibrium (DUE), and updated through the VISSIM-COM-Python interaction. Then, the output data are obtained to shape MFDs, and k-means clustering algorithm is employed to quantify the MFD-based network capacity. Finally, the methodology is implemented in a test network, and the results show the benefits of using the MFD-based method to solve the network design problem under stochastic OD demands. Specifically, the capacity paradox is also presented in the test results.

A QUADRATIC PROGRAMMING APPROACH FOR SOLVING A DYNAMIC USER EQUILIBRIUM WITH SIMULTANEOUS DEPARTURE TIME AND ROUTE CHOICE

本研究は，経路と出発時刻を同時に選択する動的利用者均衡配分 (DUE-SDR: dynamic user equilibrium with simultaneous departure time and route choice)モデルの解法を開発する．具体的には，まず，一起点多終点もしくは多起点一終点の一般ネットワークを対象とし，待ち行列の物理的長さを捨象したpoint-queueモデルを用いて渋滞を表現する枠組を示す．この枠組の下で，DUE-SDRモデルを混合線形相補性問題として定式化する．次に，こうして定式化された問題が，適当な離散時点の枠組下で等価な二次計画問題に帰着することを明らかにし，これをFrank-Wolfe法を用いて解くアルゴリズムを開発する．最後に，提案解法をSioux-Fallsネットワークに適用し，未知変数が数万個に及ぶ問題に対しても，実用的な時間内に均衡解が求められることを示す．

Perimeter gating control and citywide dynamic user equilibrium: A macroscopic modeling framework

In recent years, several perimeter control strategies have been proposed for traffic management in cities. The common factor found in these works is the use of Macroscopic Fundamental Diagram (MFD) models to describe the dynamics of the network and optimize traffic inside the perimeter by manipulating perimeter inflows. Perimeter gating control strategies are attractive for traffic management inside the inner city. However, it inevitably creates a negative impact on the traffic outside. Most of the works in this research area have neglected vehicle re-routing outside the controlled perimeter, i.e., they do not consider demand elasticity to the central region resulting from gating and the related queues. In this paper, we propose a global modeling framework capable of assessing the effect of perimeter gating control (in terms of queue, emission, and total time spent) on the full network, considering demand elasticity resulting from Dynamic User Equilibrium (DUE). Classical Proportional-Integral (PI) control scheme is used to control traffic congestion inside a central region (reservoir). The modeling framework is comprised of: (i) an accumulation-based MFD model to reproduce traffic dynamics inside the reservoir, (ii) point-queue model to represent queuing vehicles on inbound links to the gating points, and (iii) a time-dependent travel time profile based on a steady-state approximation of MFD dynamics to characterize the alternative road network (bypass). DUE is then implemented, considering instantaneous predicted travel time. This determines how the demand to the inner region is affected by the gating. The functioning of the global system is assessed by total time spent and NOx and CO2 emissions inside the reservoir and for the full network. The presented simulation results show that the perimeter gating control helps to maintain congestion at the desired level with significant improvements in the total time spent and the mean speed in the network. However, it shows a slight increase in the queues. As expected, deviation to the bypass alternative is significant and should not be neglected when carrying out a global assessment of gating system performance.