Decision Epoch Research Articles

Dynamic pickup-and-delivery for collaborative platforms with time-dependent travel and crowdshipping

We study a pickup-and-delivery problem that arises when customers randomly submit requests over the course of a day from a choice of vendors on a collaborative e-commerce portal. Based on the attributes of a customer request, a dispatcher dynamically schedules the delivery service on either a dedicated vehicle or a crowdshipper, both of whom experience time-dependent travel times. While dedicated vehicles are available throughout the day, the availability of crowdshippers is unknown a priori and they appear randomly for only portions of the day. With an objective of minimizing the sum of routing costs, piece-rate crowdshipper payments, and lateness charges, we model the uncertainty in request arrivals and crowdshipper appearances as a Markov decision process. To determine an action at each decision epoch, we employ a heuristic that partially destroys the existing routes and repairs them under the guidance of a parameterized cost function approximation that accounts for the remaining temporal capacity of delivery vehicles. We benchmark our real-time heuristic with an adaptive large neighborhood search and demonstrate the effectiveness of our method with several performance metrics. In addition, we conduct computational experiments to demonstrate the impact of inserting wait time in the route scheduling and the benefit of explicitly modeling time-dependent travel times. Through our computational testing, we also investigate the potential of demand management mechanisms that facilitate many-to-one request bundles or one-to-many request bundles to reduce the cost to service requests.

Structured replacement policies for a system subject to random mission types

AbstractThis paper optimizes condition‐based replacement policies for a mission‐oriented system. The key challenge in our problem is that the system does not work under a fixed mission type but is subject to an infinite sequence of random types of missions assigned in a Markovian manner, which is realistic in many practical situations. The mission process modulates the deterioration process. Taking advantage of the opportunities when missions are switched, condition monitoring is conducted to support replacement decision‐making. This paper considers two practical scenarios in which the type of the next mission is either available or unavailable at each decision epoch. The objective is to determine the optimal replacement decisions for both scenarios that minimize their long‐run expected average cost rates. The optimization problems are analyzed in the framework of the Markov decision process. The optimal decisions of both scenarios are proven to be of partially monotone control‐limit forms. Near‐optimal policies with multilevel thresholds are provided for more convenient decision‐making. The policy iteration algorithm is modified for efficient optimization. A numerical example is used to demonstrate the feasibility of the proposed approach.

Integrated operations strategies for shared and privately-owned autonomous vehicles: A deep reinforcement learning framework

The shared autonomous vehicle (SAV), combining shared mobility and vehicle automation, will play a pivotal role in urban transportation. Effectively matching SAVs to stochastic customer demand poses a significant challenge, particularly when addressing microscopic multi-AV coordination to prevent conflicts and congestion within a network. To tackle this issue, we propose a novel traffic control scheme called the Integrated Operation Framework for Order Matching and AV Coordination (I-OFOMAC). This scheme introduces a third-party coordinator responsible for integrating online order matching for the SAV platform and real-time AV coordination for the road manager. Our approach utilizes an offline deep reinforcement learning method to train a state value neural network, which facilitates strategic order matching. Additionally, a neural network approximating bisimulation metrics is trained to enhance exploration and accelerate learning. To achieve scalability in network-level AV coordination, we adopt a scheduled transportation scheme. The coordinator solves the I-OFOMAC during each decision epoch, optimizing order matching and AV coordination. The I-OFOMAC aims to balance the goals of minimizing AV travel delay and maximizing SAV service revenue. We conduct extensive simulation studies in the realistic Wangjing network to evaluate the proposed I-OFOMAC scheme. Compared to two-step benchmark frameworks that implement order matching and AV coordination sequentially, the I-OFOMAC achieves an average increase of 13.28% in platform revenue (ranging from 2% to 22.53%) and a 4.48% reduction in average PAV travel delay (ranging from −6.55% to 20.17%).

Online Restless Multi-Armed Bandits with Long-Term Fairness Constraints

Restless multi-armed bandits (RMAB) have been widely used to model sequential decision making problems with constraints. The decision maker (DM) aims to maximize the expected total reward over an infinite horizon under an “instantaneous activation constraint” that at most B arms can be activated at any decision epoch, where the state of each arm evolves stochastically according to a Markov decision process (MDP). However, this basic model fails to provide any fairness guarantee among arms. In this paper, we introduce RMAB-F, a new RMAB model with “long-term fairness constraints”, where the objective now is to maximize the longterm reward while a minimum long-term activation fraction for each arm must be satisfied. For the online RMAB-F setting (i.e., the underlying MDPs associated with each arm are unknown to the DM), we develop a novel reinforcement learning (RL) algorithm named Fair-UCRL. We prove that Fair-UCRL ensures probabilistic sublinear bounds on both the reward regret and the fairness violation regret. Compared with off-the-shelf RL methods, our Fair-UCRL is much more computationally efficient since it contains a novel exploitation that leverages a low-complexity index policy for making decisions. Experimental results further demonstrate the effectiveness of our Fair-UCRL.

Quantifying the Benefits of Increasing Decision-Making Frequency for Health Applications with Regular Decision Epochs

Sequential decision-making problems in the context of uncertainty naturally arise in healthcare settings. In general, the frequency at which decisions can be made or changed is determined by physical limitations, such as the frequency of doctor’s visits or transplantation offers. Quantifying the benefits of increasing the frequency of decision-making allows us to quantify the value of changing these physical constraints and thus improve the quality of care. In this paper, we study the value provided by having additional decision-making opportunities in each epoch. We model this problem using a Markov decision process (MDP) framework. We provide structural properties of the optimal policies and quantify the difference in optimal values between MDP problems of different decision-making frequencies. We analyze numerical examples using liver transplantation in high-risk patients and treatment initiation in chronic kidney disease to illustrate our findings.

Sequential learning based re-optimization approaches for less model-based dynamic pick-up routing problem

We address a lessmodel-based dynamic routing problem arising from home parcel pick-up service, where lessmodel-based means existing customers who dynamically request services independently following Poisson process with a stochastic rate. Overall, through an extended application of re-optimization (RO) strategy, a Markov decision process formulation and approximation dynamic programming- and Bayes' theorem-based solution approaches are proposed. Specifically, first a pool of basic policies corresponding to all possible values of the rate are developed offline via approximate value iteration. Then, Bayes' theorem-based sequential learning is designed that can sequentially update the belief about the rate's probability distribution over its possible values. Third, coupled with the updated belief, basic policies are collectively implemented in two different ways, resulting in RO approaches which involve constructions of two different online policies, i.e. a belief-weighted deterministic policy and a belief-based random policy, and their re-optimizations at decision epochs. In the numerical study, through comparison with model-based (i.e. using full knowledge of the rate) and model-free heuristics, our approaches are examined and valuable insights are obtained. Important insights include that (i) the belief-weighted deterministic policy outperforms the belief-based random policy, and further, (ii) the former is better than the latter at preserving the improvement resulting from the improved model-based policy.

Linear Program-Based Policies for Restless Bandits: Necessary and Sufficient Conditions for (Exponentially Fast) Asymptotic Optimality

We provide a framework to analyze control policies for the restless Markovian bandit model under both finite and infinite time horizons. We show that when the population of arms goes to infinity, the value of the optimal control policy converges to the solution of a linear program (LP). We provide necessary and sufficient conditions for a generic control policy to be (i) asymptotically optimal, (ii) asymptotically optimal with square root convergence rate, and (iii) asymptotically optimal with exponential rate. We then construct the LP-index policy that is asymptotically optimal with square root convergence rate on all models and with exponential rate if the model is nondegenerate in finite horizon and satisfies a uniform global attractor property in infinite horizon. We next define the LP-update policy, which is essentially a repeated LP-index policy that solves a new LP at each decision epoch. We conclude by providing numerical experiments to compare the efficiency of different LP-based policies. Funding: This work was supported by Agence Nationale de la Recherche [Grant ANR-19-CE23-0015].

Non‐periodic inspection and replacement policy of system subject to non‐homogeneous gamma degradation process

AbstractThe non‐homogeneous gamma process can exhibit a convex average trend with a non‐linear shape parameter, providing significant flexibility in fitting an accelerated degradation process. However, few studies have been conducted on the maintenance modeling of systems undergoing a non‐homogeneous gamma degradation process despite its versatility. Meanwhile, most maintenance studies adopt a periodic inspection policy to reveal the degradation level. However, it is reasonable to inspect less frequently when the degradation level of the system is low and to perform inspections more frequently when the degradation level increases. Therefore, we conduct a study on the inspection/replacement policy of systems undergoing a non‐homogeneous gamma degradation process, where three non‐periodic inspection policies are investigated. We formulate the problem as a SMDP framework and present a value iteration algorithm to obtain the optimal action at each decision epoch that minimizes the long‐term average cost. A comprehensive numerical study is provided to illustrate the effectiveness of our proposed model.

Dynamic inspection and maintenance scheduling for multi-state systems under time-varying demand: Proximal policy optimization

Inspection and maintenance activities are effective ways to reveal and restore the health conditions of many industrial systems, respectively. Most extant works on inspection and maintenance optimization problems have assumed that systems operate under a time-invariant demand. Such a simplified assumption is oftentimes violated by a changeable market environment, seasonal factors, and even unexpected emergencies. In this article, with the aim of minimizing the expected total cost associated with inspections, maintenance, and unsupplied demand, a dynamic inspection and maintenance scheduling model is proposed for Multi-State Systems (MSSs) under a time-varying demand. Non-periodic inspections are performed on the components of an MSS and imperfect maintenance actions are dynamically scheduled based on the inspection results. By introducing the concept of decision epochs, the resulting inspection and maintenance scheduling problem is formulated as a Markov Decision Process (MDP). The Deep Reinforcement Learning (DRL) method with a Proximal Policy Optimization (PPO) algorithm is customized to cope with the “curse of dimensionality” of the resulting sequential decision problem. As an extra input feature for the agent, the category of decision epochs is formulated to improve the effectiveness of the customized DRL method. A six-component MSS, along with a multi-state coal transportation system, is given to demonstrate the effectiveness of the proposed method.

The pickup and delivery problem with multiple depots and dynamic occasional drivers in crowdshipping delivery

In recent years, crowdshipping has gained much popularity because it not only has economic and environmental advantages, but also is regarded as an effective way to increase the resilience of supply chains and logistics. In this work, we focus on a last-mile delivery system that supports its O2O e-commerce partner for processing and fulfilling online orders in an urban area. Besides the full-time regular drivers, the system also relies on occasional drivers from society who dynamically announce their availability over the time horizon. We model the studied problem as a multi-depot pickup and delivery problem with dynamic occasional drivers. An event-based rolling horizon solution approach is proposed to solve the problem through iteratively calling for a re-optimization procedure to update the delivery plan at each decision epoch. Besides, we investigate the scenarios of disruptions to the delivery system. Extensive numerical experiments have proven the effectiveness of our proposed approach and have also demonstrated the studied crowdshipping system can help achieve the supply chain and logistics resilience. Moreover, sensitivity analysis on several important parameters have been carried out and managerial insights are drawn.

A dynamic approach for the multi-compartment vehicle routing problem in waste management

Urban areas worldwide face a significant environmental challenge which is increasing municipal solid waste rate. Addressing its negative consequences necessitates advancements in waste management systems. Although the previous research focused on the static routing approach in the collection phase, this paper adds a dynamic municipal solid waste collection scheme to optimize vehicle routing, accounting for fluctuations in waste generation and changes in transportation systems. This study employs, for the first time, the application of a discrete choice model (DCM) to streamline the process of re-optimization in dynamic vehicle routing problems (DVRP). At each decision epoch, DCM is applied to determine the likelihood of choosing the next geographical zone to visit bins based on current waste generation levels and traveling costs. Moreover, the multi-compartment vehicles are considered to preserve waste segregation during transportation, thereby increasing operational efficiency and regulatory compliance. Another contribution of this paper is to determine visiting priority for each bin by adjusting the time window based on the threshold waste level. Hence, this paper proposes a framework for sustainable, efficient, and effective waste management practices by integrating the benefits of dynamic and multi-compartment routing. Furthermore, a hybrid Genetic and Particle Swarm Optimization algorithm has been designed to find the best solution for the studied problem as well as some of the latest and most proficient metaheuristic algorithms. Finally, the Best Worst Method is applied to find the best-proposed algorithm to solve the presented problem, indicating that the hybrid algorithm has the highest performance in providing high-quality route plans.

Off-line approximate dynamic programming for the vehicle routing problem with a highly variable customer basis and stochastic demands

We study a stochastic variant of the vehicle routing problem (VRP) arising in the context of domestic donor collection services. The problem we consider combines the following attributes. Customers requesting services are variable, in the sense that they are stochastic, but are not restricted to a predefined set. Furthermore, demand volumes are also stochastic and are observed upon visiting customers. The objective is to maximize the expected served demands while meeting vehicle capacity and time restrictions. We call this problem the VRP with a highly Variable Customer basis and Stochastic Demands (VRP-VCSD). We first propose a classical Markov Decision Process (MDP) formulation for the VRP-VCSD. The resulting model is, however, unusable due to the explosion in the dimension of the state and action spaces.To solve the VRP-VCSD, we propose a number of methodological contributions aimed at reducing the state and action spaces. We first reformulate the MDP as an MDP with a consecutive action selection procedure. In this formulation, we enforce the treatment of a single vehicle (as opposed to multiple vehicles) at each decision epoch. We then introduce an observation function that selects a subset of the available information, which is deemed relevant for the considered vehicle in each epoch. We develop a Q-learning algorithm called QN-CO. In particular, we use a continuous state representation and incorporate a two-layer artificial neural network to approximate the Q values. Furthermore, we propose an aggregation strategy yielding a fixed-size output. Finally, we enhance our algorithm with Replay Memory and a Double Q Network.We conduct a thorough computational analysis. Results show that QN-CO considerably outperforms five benchmark policies. Moreover, we show that QN-CO can compete with specialized methods developed for the particular case of the VRP-VCSD where customer locations and expected demands are known in advance.

Joint Optimization of Freshness and Fidelity for Status Updates in a Periodical Decision System

Development of applications in real-time service promotes more stringent freshness and fidelity requirements for the status update and decision systems. However, the freshness and fidelity of status updates are mutually restrictive, as fresh information requires timely transmission whereas high quantization precision incurs long transmission time. This work improves trade-off between the freshness and fidelity of status updates at periodical decision epochs by selecting appropriate transmission time and quantization precision. Specifically, the age of information (AoI) metric is used to assess data freshness, while distortion that consists of both static and dynamic parts is used to characterize the fidelity of status updates. We first formulate a joint AoI and distortion optimization problem. Then, the optimal transmission time and quantization precision are theoretically analyzed. We further minimize the sum of AoI and distortion by using power allocation strategy which selects a proper budget SNR of the wireless channel. Numerical results show that by using the optimized transmission time, quantization precision, associated with transmit power allocation, the overall freshness and fidelity of status updates can be improved remarkably.

A learning-based scheme for channel allocation to vehicular users in wireless networks

Resource allocation algorithms in wireless networks can require solving complex optimization problems at every decision epoch. For large scale networks, when decisions need to be taken on time scales of milliseconds, using standard convex optimization solvers for computing the optimum can be a time-consuming affair that may impair real-time decision making. In this paper, we propose to use Data-driven and Deep Feedforward Neural Networks (DFNN) for learning the relation between the inputs and the outputs of two such resource allocation algorithms that were proposed in Nguyen et al. (2019, 2020). On numerical examples with realistic mobility patterns, we show that the learning algorithm yields an approximate yet satisfactory solution with much less computation time.

Minimizing Age-Upon-Decisions in Bufferless System: Service Scheduling and Decision Interval

The data freshness at decision epochs of time-sensitive applications, e.g., auto-driving vehicles and autonomous underwater robots, is jointly affected by the statistics of update process and decision process. This work considers an update-and-decision system with a Poisson-arrival bufferless queue, where updates are delivered and processed for making decisions with exponential or periodic intervals. We use age-upon-decisions (AuD) to characterize timeliness of updates at decision moments, and the missing probability to specify whether updates are useful for decision-making. Our theoretical analyses 1) present the average AuDs and the missing probabilities for bufferless systems with exponential or deterministic decision intervals under different service time distributions; 2) show that for service scheduling, the deterministic service time achieves a lower average AuD and a smaller missing probability than the uniformly distributed and the negative exponentially distributed service time; 3) prove that the average AuD of periodical decision system is larger than and will eventually drop to that of Poisson decision system along with the increase of decision rate; however, the missing probability in periodical decision system is smaller than that of Poisson decision system. The numerical results and simulations verify the correctness of our analyses, and demonstrate that the bufferless systems outperform the systems applying infinite buffer size.

Inspection strategies for quality products with rewards in a multi-stage production

In a multi-stage manufacturing system, defective components are generated due to deteriorating machine parts and failure to install the feed load. In these circumstances, the system requires inspection counters to distinguish imperfect items and takes a few discreet decisions to produce impeccable items. Whereas the prioritisation of employee appreciation and working on reward is one of the important policies to improve productivity. Here we look at the multi-stage manufacturing system as an M/PH/1 queue model and rewards are given for using certain inspection strategies to produce the quality items. A matrix analytical method is proposed to explain a continuous-time Markov process in which the reward points are given to the strategy of inspection in each state of the system. By constructing the value functions of this dynamic programming model, we derive the optimal policy and the optimal average reward of the entire system in the long run. In addition, we obtain the percentage of time spent on each system state for the probability of conformity and non-conformity of the product over the long term. The results of our computational experiments and case study suggest that the average reward increases due to the actions are taken at each decision epoch for rework and disposal of the non-conformity items.

A rolling-horizon decision framework for integrating relocation and user flexibility in one-way electric carsharing systems

One-way electric carsharing system has become the most widely used form of carsharing service in practice. However, the demand uncertainty, car imbalance, parking shortage, and battery charging jointly impose great challenges upon the operation. Many carsharing operators, e.g., GoFun, offer incentives to flexible users for accepting alternative itineraries and also adopt a full reservation policy. In this paper, we propose a rolling-horizon decision framework that integrates the operator’s car and staff relocation decisions, user flexibility, and charging of electric cars in the real-time setting. We formulate the problem at each decision epoch as a time–space network flow model that simultaneously optimizes the decisions. We develop an iterated local search heuristic algorithm to solve the model for large-scale practical problems. Based on theoretical insights and clustering of stations, we develop new local search procedures customized for the dynamic carsharing problem. We use real-world data from GoFun as an example to conduct extensive computational experiments. The results show that our algorithm significantly outperforms a particle swarm optimization algorithm and the popular greedy policy, and also has a low efficiency loss compared to the off-line heuristic solution. The transportation network size, clustering size, and operational efficiency are revealed to have significant impacts on the profit and service level.

Community logistics for dynamic vehicle dispatching: The effects of community departure “time” and “space”

The rise of e-commerce has completely transformed the dynamicity and problem nature of last-mile delivery, owing to significant B2C customer demand in compact urban areas. Given the unprecedented growths of urban last-mile deliveries, this paper proposes a novel postponement prioritized-route optional approach, namely Community Logistics (CL), as a new logistics tool to manage dynamic arrivals of delivery requests received in e-commerce hubs. Each vehicle is responsible for serving a “community”. At each decision epoch, fragmented e-commerce delivery requests arrived at the depot are either allocated to a community or postponed to later epochs for actual last-mile delivery. With an objective of consolidating newly arrived requests, we develop two dynamic policies – temporal and spatial, respectively for temporally delaying vehicle’s departure and spatially allocating more pre-partitioned geographical cells into one community. The main contribution of this study lies in a spatiotemporal relativity analysis and a comparative analysis. The former demonstrates the essence of incorporating both dynamic community departure times and dynamic community regions into managing urban e-commerce deliveries, whereas the latter validates the merits of Community Logistics against dynamic vehicle routing solutions. In the end, we call for further developments of community logistics strategies to address the impacts of urban deliveries due to the rise of e-commerce and online shopping.

A column generation heuristic for the dynamic bicycle rebalancing problem

Public bicycle sharing systems are becoming an essential part of the future urban mobility system. Real-time monitoring of the system state through sensors on bicycles and/or stations gives possibilities for advanced coordination of the system. In this paper, we consider the dynamic bicycle rebalancing problem, where bicycles are re-positioned by service vehicles to prevent stations from becoming completely full or empty, and so satisfying the demand for bicycles or locks. We solve the problem in a rolling horizon fashion with dynamic deterministic bicycle rebalancing subproblems (DDBRS) at the decision epochs. To solve the DDBRS within a few seconds in real-time, we propose a novel column generation heuristic (CGH). The CGH is tested within a simulation framework based on real data from the bicycle sharing system in Oslo. We show that the CGH is able to solve large real-life instances with computational times that are suitable for actual operation and that it provides significantly improved solutions compared with current planning practice. We also perform a number of tests to analyze the effect of changing the number of bicycles and locks in the system, as well as adding extra service vehicles. The case company is now making preparations to implement an optimization-based decision support system based on the CGH proposed in this paper.

Optimal Admission Control for Multiclass Queues with Time-Varying Arrival Rates via State Abstraction

We consider a novel queuing problem where the decision-maker must choose to accept or reject randomly arriving tasks into a no buffer queue which are processed by N identical servers. Each task has a price, which is a positive real number, and a class. Each class of task has a different price distribution, service rate, and arrives according to an inhomogenous Poisson process. The objective is to decide which tasks to accept so that the total price of tasks processed is maximised over a finite horizon. We formulate the problem using a discrete time Markov Decision Process (MDP) with a hybrid state space. We show that the optimal value function has a specific structure, which enables us to solve the hybrid MDP exactly. Moreover, we rigorously prove that as the gap between successive decision epochs grows smaller, the discrete time solution approaches the optimal solution to the original continuous time problem. To improve the scalability of our approach to a greater number of servers and task classes, we present an approximation based on state abstraction. We validate our approach on synthetic data, as well as a real financial fraud data set, which is the motivating application for this work.