Greedy Policy Research Articles

Reinforcement learning-based automated modulation switching algorithm for an enhanced underwater acoustic communication

Acoustic communication is the preferred method for underwater communication due to its superior propagation characteristics, including longer range, lower attenuation, better penetration, and adaptability to water density. However, challenges like varying water depths, temperature gradients, noise, and multipath propagation require innovative solutions beyond fixed-data rate systems and single modulation schemes. To address these challenges, we propose a reinforcement learning (RL)-based automated modulation switching algorithm designed to enhance data transmission efficiency. This approach leverages the UNETStack software and the Frequency Hopping-Binary Frequency Shift Key (FH-BFSK) modulation, extending its capabilities by incorporating Amplitude Shift Keying (ASK), Phase Shift Keying (PSK), and Orthogonal Frequency Division Multiplexing (OFDM) modulation techniques. We designed a cost-effective, software-controlled acoustic modem using commercial off-the-shelf (COTS) components, enabling full-duplex underwater communication. The RL algorithm utilizes a Q-matrix guided by a greedy policy to assess network conditions, such as data volume and data rates. It continuously monitors underwater environments to select the most suitable modulation scheme dynamically. Experimental validation demonstrates a 3.648 % improvement in Received Signal Strength Indicator (RSSI), a 32 % reduction in bit error rate at a constant 7 dB Signal-to-Noise Ratio (SNR), and a 5 % increase in utility at 10 dB SNR when using OFDM selected by the reinforcement learning algorithm, compared to FH-BFSK.

Load-aware switch migration for controller load balancing in edge–cloud architectures

As the fundamental infrastructure for edge–cloud architectures, the inter-datacenter elastic optical network is used for data analysis and processing. As the demand for applications increases, the large number of service requests increases the processing overhead in the control plane, resulting in unbalanced controller loads. Existing switch migration mechanisms have been proposed for controller load balancing. Unfortunately, most of the existing mechanisms only consider the switch with the highest flow request rate as the migration object in the process of switch selection, and ignore the migration cost generated in the switch migration activity, such as the update cost of flow request message and the deployment cost of migration rule, which may increase the controller load. Additionally, most of them choose the controller with light load as the target controller to associate with the switch to be migrated, without considering whether the target controller is overloaded after the switch migration, which leads to the low load balancing performance of the controller. In view of the above problems, this paper proposes a Load-Aware Switch Migration (LASM) mechanism in edge–cloud architectures. The LASM mechanism models and analyses the cost metrics affecting switch migration and selects lower-cost switches from overloaded controller-controlled domain networks for migration activities. Besides, the LASM mechanism models switch migration based on the 0-1 knapsack problem and avoids overloading the target controllers through a greedy policy to achieving optimal migration activities. The experimental results show that the proposed LASM mechanism improves controller load balancing performance by an average of 34.3%, eliminates migration costs by 30.2%, and reduces response times by an average of 39.3%, respectively, compared to existing solutions.

Performance paradox of dynamic matching models under greedy policies

We consider the stochastic matching model on a non-bipartite compatibility graph and analyze the impact of adding an edge to the expected number of items in the system. One may see adding an edge as increasing the flexibility of the system, for example, asking a family registering for social housing to list fewer requirements in order to be compatible with more housing units. Therefore, it may be natural to think that adding edges to the compatibility graph will lead to a decrease in the expected number of items in the system and the waiting time to be assigned. In our previous work, we proved this is not always true for the First Come First Matched discipline and provided sufficient conditions for the existence of the performance paradox: despite a new edge in the compatibility graph, the expected total number of items can increase. These sufficient conditions are related to the heavy-traffic assumptions in queueing systems. The intuition behind this is that the performance paradox occurs when the added edge in the compatibility graph disrupts the draining of a bottleneck. In this paper, we generalize this performance paradox result to a family of so-called greedy matching policies and explore the type of compatibility graphs where such a paradox occurs. Intuitively, a greedy matching policy never leaves compatible items unassigned, so the state space of the system consists of finite words of item classes that belong to an independent set of the compatibility graph. Some examples of greedy matching policies are First Come First Match, Match the Longest, Match the Shortest, Random, Priority. We prove several results about the existence of performance paradoxes for greedy disciplines for some family of graphs. More precisely, we prove several results about the lifting of the paradox from one graph to another one. For a certain family of graphs, we prove that there exists a paradox for the whole family of greedy policies. Most of these results are based on strong aggregation of Markov chains and graph theoretical properties.

Surgery scheduling in flexible operating rooms by using a convex surrogate model of second-stage costs

We study the elective surgery planning problem in a hospital with operating rooms shared by elective and emergency patients. This problem is split in two distinct phases. First, a subset of patients to be operated in the next planning period is selected and the selected patients are assigned to a block and a tentative starting time. Then, in the online phase of the problem, a policy decides how to insert the emergency patients in the schedule and may cancel planned surgeries. The overall goal is to minimize the expectation of a cost function representing the assignment of patient to blocks, case cancellations, overtime, waiting time and idle time. We model the offline problem by a two-stage stochastic program, and show that the optimal second-stage costs can be approximated by a convex piecewise linear surrogate model that can be computed in a preprocessing step. This results in a mixed integer program which can be solved very fast, even for large instances of the problem. We also describe a greedy policy for the online phase of the problem, and analyze the performance of our approach by comparing it to both heuristic methods or approaches relying on sampling average approximation (SAA) on a large set of benchmarking instances. Our simulations indicate that our approach can reduce the expected costs by as much as 30% compared to heuristic methods and it can solve problems with 1000 patients in about one minute, while SAA-approaches fail to obtain good solutions within 30 min on small instances.

A parking incentive allocation problem for ridesharing systems

Ridesharing occurs when people with similar schedules and itineraries travel together to reduce their commuting costs. In this paper, we study how parking spaces can be used to incentivize drivers to participate in ridesharing systems. We develop a Parking Incentive Allocation (PIA) system to promote and allocate parking lots to ridesharing drivers in a stochastic and dynamic environment. The optimization problem is formulated at each period as a multi-stage stochastic decision-dependent program. To overcome the complexity of the model, we propose one greedy policy, and three approximations including two stochastic policies and an expected-value policy. We evaluate the effectiveness of the four policies on the data generated from GPS information collected by the MTL Trajet project, which studies residents’ travel patterns throughout the city of Montreal. The computational results indicate that on average, the approximate policies can improve the total distance saving by more than 20% over various problem settings. Additionally, the results show that the performance of the PIA system is significantly influenced by the attractiveness of the parking incentive to drivers.

Optimal Policies and Heuristics to Match Supply with Demand for Online Retailing

Problem definition: We consider an online retailer selling multiple products to different zones over a finite horizon with multiple periods. At the start of the horizon, the retailer orders the products from a single supplier and stores them at multiple warehouses. The retailer determines the products’ order quantities and their storage quantities at each warehouse subject to its capacity constraint. At the end of each period, after random demands in the period are realized, the retailer chooses the retrieval quantities from each warehouse to fulfill the demands of each zone. The objective is to maximize the retailer’s expected profit over the finite horizon. Methodology/results: For the single-zone case, we show that the multiperiod problem is equivalent to a single-period problem and the optimal retrieval decisions follow a greedy policy that retrieves products from the lowest-cost warehouse. We design a nongreedy algorithm to find the optimal storage policy, which preserves a nested property: Among all nonempty warehouses, a smaller-index warehouse contains all the products stored in a larger-index warehouse. We also analytically characterize the optimal ordering policy. The multizone case is unfortunately intractable analytically, and we propose an efficient heuristic to solve it, which involves a nontrivial hybrid of three approximations. This hybrid heuristic outperforms two conventional benchmarks by up to 22.5% and 3.5% in our numerical experiments with various horizon lengths, fulfillment frequencies, warehouse capacities, demand variations, and demand correlations. Managerial implications: A case study based on data from a major fashion online retailer in Asia confirms the superiority of the hybrid heuristic. With delicate optimization, the heuristic improves the average profit by up to 16% compared with a dedicated policy adopted by the retailer. The hybrid heuristic continues to outperform the benchmarks for larger networks with various structures. Funding: X. Li is supported by the Singapore Ministry of Education [Tier 1 Grant 23-0619-P0001]. Y. F. Lim is grateful for the support from the Singapore Management University under the Maritime and Port Authority Research Fellowship and the Singapore Ministry of Education [Tier 1 Grant MSS23B001]. Supplemental Material: The online appendix is available at https://doi.org/10.1287/msom.2021.0394 .

Dynamic vehicle routing problem with drone resupply for same-day delivery

We study a same-day delivery problem where customer orders arrive dynamically throughout the day and the service operator must determine, in real-time, whether to accept the orders and how to adjust the ongoing distribution plan. We develop a route-based Markov Decision Process and an efficient online policy to dynamically route a truck that can receive newly arrived orders along its route via drones dispatched from a depot. Numerical experiments show that our online policy has an average fill rate decrease of at most 20% over the perfect-information counterpart. Further, this online policy has a fill rate increase of up to 8% over a naïve greedy policy. We also show that drone resupply increases fill rates by up to 21% compared to a conventional truck-only resupply system. Computational times to make each decision are in the hundredths of a second, thus allowing real-time feedback to customers regarding their eligibility for same-day delivery.

Finite-Time Frequentist Regret Bounds of Multi-Agent Thompson Sampling on Sparse Hypergraphs

We study the multi-agent multi-armed bandit (MAMAB) problem, where agents are factored into overlapping groups. Each group represents a hyperedge, forming a hypergraph over the agents. At each round of interaction, the learner pulls a joint arm (composed of individual arms for each agent) and receives a reward according to the hypergraph structure. Specifically, we assume there is a local reward for each hyperedge, and the reward of the joint arm is the sum of these local rewards. Previous work introduced the multi-agent Thompson sampling (MATS) algorithm and derived a Bayesian regret bound. However, it remains an open problem how to derive a frequentist regret bound for Thompson sampling in this multi-agent setting. To address these issues, we propose an efficient variant of MATS, the epsilon-exploring Multi-Agent Thompson Sampling (eps-MATS) algorithm, which performs MATS exploration with probability epsilon while adopts a greedy policy otherwise. We prove that eps-MATS achieves a worst-case frequentist regret bound that is sublinear in both the time horizon and the local arm size. We also derive a lower bound for this setting, which implies our frequentist regret upper bound is optimal up to constant and logarithm terms, when the hypergraph is sufficiently sparse. Thorough experiments on standard MAMAB problems demonstrate the superior performance and the improved computational efficiency of eps-MATS compared with existing algorithms in the same setting.

New Classes of the Greedy-Applicable Arm Feature Distributions in the Sparse Linear Bandit Problem

We consider the sparse contextual bandit problem where arm feature affects reward through the inner product of sparse parameters. Recent studies have developed sparsity-agnostic algorithms based on the greedy arm selection policy. However, the analysis of these algorithms requires strong assumptions on the arm feature distribution to ensure that the greedily selected samples are sufficiently diverse; One of the most common assumptions, relaxed symmetry, imposes approximate origin-symmetry on the distribution, which cannot allow distributions that has origin-asymmetric support. In this paper, we show that the greedy algorithm is applicable to a wider range of the arm feature distributions from two aspects. Firstly, we show that a mixture distribution that has a greedy-applicable component is also greedy-applicable. Second, we propose new distribution classes, related to Gaussian mixture, discrete, and radial distribution, for which the sample diversity is guaranteed. The proposed classes can describe distributions with origin-asymmetric support and, in conjunction with the first claim, provide theoretical guarantees of the greedy policy for a very wide range of the arm feature distributions.

Delay-sensitive energy-efficient routing scheme for the Wireless Sensor Network with path-constrained mobile sink

The use of mobile sinks (MSs) can address several challenges for traditional wireless sensor networks (WSNs), such as energy holes, meanwhile it also introduces some new issues. Especially, for the path constraint MS, the fixed trajectory and uncontrollable moving speed results in significant delay and even buffer overflows. Existing research mainly focused on delay-tolerant applications which are unsuitable for delay-sensitive data or adopted the greedy policy to collect data which is inefficient in energy. To minimize the energy consumption and guarantee delay requirement simultaneously, we propose a delay-sensitive and energy-efficient routing scheme. Firstly, to obtain the MS position, we propose a K-nearest neighbor-based prediction model, which can reduce the energy and delay overhead of the location update. Based on the prediction result, an efficient cluster routing scheme is designed. To ensure both search speed and the optimality of the cluster head (CH), we improve the Reed-Deer algorithm (RDA) as IRDA, and use the IRDA to find out CHs. To remove the redundancy of the CH, a removal method is proposed. After constructing clusters, a method is used to determine the next-hop of the CH by using the predication position of the MS, the received timestamp and the delay requirement of the data. A fault tolerance mechanism is supplied to deal the predication error and other unexpected situations. By applying IRDA and the prediction model, the proposed routing scheme avoids the frequent construction of new routes, reduces the hops of the end-to-end path, minimizes network energy consumption, and ensures timely data transmission. The comparison results indicates that the proposed protocol can effectively improve the network performance while guarantee delay requirement. Especially in realistic scenario, the proposed algorithm can effectively reduce energy consumption and the end-to-end delay and improve the packet delivery ratio.

Application of artificial intelligence based on the fuzzy control algorithm in enterprise innovation

Artificial Intelligence (AI) has gained immense popularity in recent years as many enterprises have realized their potential to change the way of conducting business innovatively. The new concepts, items, or procedures are developed and implemented within a business or organization to enhance productivity, effectiveness, and competitiveness, and this is called Enterprise Innovation (EI). AI techniques are required to make decisions more effectively in challenging and dynamic situations, like EI, as result of competitive marketplace. Hence, an intelligent, innovative strategy with Q-learning and Takagi Sugeno Fuzzy Control (Q-TSFC) algorithm has been proposed as it combines adaptive learning and Fuzzy Logic (FL) that humans understand to improve decision-making in enterprise innovation. Q-learning seeks to maximize the enterprise's profit by utilizing the newly acquired knowledge, exploring activities, and adaptive learning based on the optimal ε greedy policy that results with rewards and the experiences. To develop a framework for making decisions and connections between input from the learned Q values and output decisions using enterprise expertise and linguistic conventions. The objective is to handle language uncertainty and imprecision in market trend. So, it leads to right decisions even without accurate numerical facts. The proposed approach is validated by evaluating metrics like cost savings, customer satisfaction, and innovation performance efficiency in the competitive edge in the market. With this proposed Q-TSFC algorithm, the obtained results are 96.5% customer satisfaction ratio, 96% enterprise performance efficiency, cost savings of about 48% profitable value, and the coefficient of determination R2 is 0.83, respectively.

Which Link Matters? Maintaining Connectivity of Uncertain Networks Under Adversarial Attack

This paper studies connectivity maintenance in uncertain networks under adversarial attack, where a defender conceals crucial links to prevent the largest connected component from being decomposed by an attacker. In contrast with its static counterpart, connectivity maintenance in uncertain networks involves additional probing on links to determine their existence. Therefore, by modeling an uncertain network as a random graph with each link associated with an existence probability and a probing cost, our goal is to design a defensive strategy for link selection that maximizes the expected size of the largest remaining connected component with the minimum expected probing cost, and moreover, the strategy should be independent of the attacking patterns. To this end, we first unravel the computational complexity of the problem by proving its NP-hardness, and then propose optimal defensive strategies based on dynamic programming and multi-objective optimization. Due to the prohibitive computational cost of optimality, two approximate defensive strategies are further designed to pursue decent performance with quasilinear complexity, in which the first one is a heuristic approach that quantifies the link vulnerability through an analogy from the degree centrality of a vertex in static networks to the connectivity weight of a link in uncertain networks, and the second one is an adaptive greedy policy incorporating the minimax rule from game theory, which minimizes the possible loss suffered by the defender in a worst-case scenario and has a constant approximation ratio. Extensive experiments on both synthetic and real-world network datasets under diverse attacking patterns demonstrate the superiority of the proposed strategies over baselines.

Sequential Search with Acquisition Uncertainty

We study a variation of the classical Pandora’s problem in which a decision maker (DM) sequentially explores alternatives from a given set and learns their values while trying to acquire the best alternative. The variations in the model we study are (i) alternatives randomly become unavailable during exploration and (ii) the DM’s ability to acquire a remaining alternative is uncertain and depends on a chosen offer price. Such acquisition uncertainties arise in many applications, including housing search, hiring problems, and e-commerce, but greatly complicate the search problem in that optimal policies retain all previously explored alternative values as part of the problem state, as opposed to only the highest explored value as in Pandora’s rule. Our central insight is that despite the complexity that these acquisition uncertainties create, simple greedy policies based on static sequencing and a single threshold value enjoy strong performance guarantees. We develop such a class of policies and show how to compute them using a greedy algorithm whose worst-case run-time scales linearly (up to logarithmic factors) in the number of alternative types. We show that our policies (a) are asymptotically optimal in high multiplicity regimes with many alternatives and (b) obtain at least [Formula: see text] of the optimal value under a broad set of conditions. Extensive numerical examples support this theory: We illustrate our policies on a variation of Pandora’s problem with disappearing alternatives and housing search on models calibrated on data from the online brokerage Redfin. In these examples, our policies significantly outperform policies based on Pandora’s rule. This paper was accepted by Omar Besbes, revenue management and market analytics. Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2022.00203 .

Sampling Efficient Deep Reinforcement Learning Through Preference-Guided Stochastic Exploration.

Stochastic exploration is the key to the success of the deep Q -network (DQN) algorithm. However, most existing stochastic exploration approaches either explore actions heuristically regardless of their Q values or couple the sampling with Q values, which inevitably introduce bias into the learning process. In this article, we propose a novel preference-guided ϵ -greedy exploration algorithm that can efficiently facilitate exploration for DQN without introducing additional bias. Specifically, we design a dual architecture consisting of two branches, one of which is a copy of DQN, namely, the Q branch. The other branch, which we call the preference branch, learns the action preference that the DQN implicitly follows. We theoretically prove that the policy improvement theorem holds for the preference-guided ϵ -greedy policy and experimentally show that the inferred action preference distribution aligns with the landscape of corresponding Q values. Intuitively, the preference-guided ϵ -greedy exploration motivates the DQN agent to take diverse actions, so that actions with larger Q values can be sampled more frequently, and those with smaller Q values still have a chance to be explored, thus encouraging the exploration. We comprehensively evaluate the proposed method by benchmarking it with well-known DQN variants in nine different environments. Extensive results confirm the superiority of our proposed method in terms of performance and convergence speed.

Approximate Policy Iteration With Deep Minimax Average Bellman Error Minimization.

In this work, we investigate the utilization of deep approximate policy iteration (DAPI) in estimating the optimal action-value function Q* within the context of reinforcement learning, employing rectified linear unit (ReLU) ResNet as the underlying framework. The iterative process of DAPI incorporates the minimax average Bellman error minimization principle. It employs ReLU ResNet to estimate the fixed point of the Bellman equation, which is aligned with the estimated greedy policy. Through error propagation, we derive nonasymptotic error bounds between Q* and the estimated Q function induced by the output greedy policy in DAPI. To effectively control the Bellman residual error, we address both the statistical and approximation errors associated with the α -mixing dependent data derived from Markov decision processes, using the techniques of empirical process and deep approximation theory, respectively. Furthermore, we present a novel generalization bound for ReLU ResNet in the presence of dependent data, as well as an approximation bound for ReLU ResNet within the Hölder class. Notably, this approximation bound contributes to a significant improvement in the dependence on the ambient dimension, transitioning from an exponential relationship to a polynomial one. The derived nonasymptotic error bounds explicitly depend on factors such as the sample size, the ambient dimension (in polynomial terms), and the width and depth of the neural networks. Consequently, these bounds serve as valuable theoretical guidelines for appropriately setting the hyperparameters, thereby enabling the achievement of the desired convergence rate during the training process of DAPI.

A computation offloading method with distributed double deep Q‐network for connected vehicle platooning with vehicle‐to‐infrastructure communications

AbstractCurrent connected vehicle applications, such as platooning require heavy‐load computing capability. Although mobile edge computing (MEC) servers connected to the roadside intelligence facility can assist such separable applications from vehicles, it is a challenge to coordinate the allocation of subtasks among vehicles and MEC servers on the premise of ensuring communication quality. Therefore, an offloading algorithm is proposed based on a double deep Q‐network to solve the placement of subtasks for vehicle to infrastructure and vehicle to vehicle cases. This algorithm considers the randomness of task generation and is model‐free. The MEC server can assist the vehicle in training the neural network and storing relevant state transitions. To improve the performance of the algorithm, the decaying greedy policy is incorporated for faster convergence. The simulation results showed that this algorithm performed well in reducing the dropped subtask rate, average time delay, and total energy consumption.

A Real-Time Control Policy to Achieve Maximum Throughput of an Online Order Fulfillment Network

Several major companies operate large online order fulfillment systems to ship goods from fulfillment centers through a distribution network to customer destinations in response to purchase orders. These networks make several types of decisions in real-time to serve customers. First, when a customer places an order, when and where (which fulfillment center) is it fulfilled from? Second, once an order has been packaged, how is it moved through the network to get to the customer? Making optimal decisions can yield significant cost savings or improvements in customer service. Unfortunately, these are large optimization problems, and are furthermore subject to uncertainty in the products and destinations of customer orders and the inventory replenishment of the fulfillment centers. This uncertainty makes the problem difficult to solve to optimality. Although the problem can be modeled as a Markov decision process, solving it exactly using standard computational methods is not possible due to the curse of dimensionality. We propose an alternative approach to this problem. We define a relatively simple real-time control policy and prove that it serves all customer demand if at all possible. This proof is achieved using Lyapunov drift techniques to relate the real-time control performance to the average performance necessary to serve all customers on average. Correspondingly, we characterize the average network performance, which may be used for network topology design while the control policy adapts to real-time stochasticity. We demonstrate the performance and stability properties on a numerical example based on hundreds of Amazon facility locations in the United States. The max-pressure control and greedy policies perform similarly at low demands, but at higher demand the throughput properties of the max-pressure control manifest as improvements in throughput and customer service metrics. Funding: Financial support from the National Science Foundation [Grant 1935514] is gratefully acknowledged. Supplemental Material: The e-companion is available at https://doi.org/10.1287/trsc.2023.0096 .

Task Scheduling for Federated Learning in Edge Cloud Computing Environments by Using Adaptive-Greedy Dingo Optimization Algorithm and Binary Salp Swarm Algorithm

With the development of computationally intensive applications, the demand for edge cloud computing systems has increased, creating significant challenges for edge cloud computing networks. In this paper, we consider a simple three-tier computational model for multiuser mobile edge computing (MEC) and introduce two major problems of task scheduling for federated learning in MEC environments: (1) the transmission power allocation (PA) problem, and (2) the dual decision-making problems of joint request offloading and computational resource scheduling (JRORS). At the same time, we factor in server pricing and task completion, in order to improve the user-friendliness and fairness in scheduling decisions. The solving of these problems simultaneously ensures both scheduling efficiency and system quality of service (QoS), to achieve a balance between efficiency and user satisfaction. Then, we propose an adaptive greedy dingo optimization algorithm (AGDOA) based on greedy policies and parameter adaptation to solve the PA problem and construct a binary salp swarm algorithm (BSSA) that introduces binary coding to solve the discrete JRORS problem. Finally, simulations were conducted to verify the better performance compared to the traditional algorithms. The proposed algorithm improved the convergence speed of the algorithm in terms of scheduling efficiency, improved the system response rate, and found solutions with a lower energy consumption. In addition, the search results had a higher fairness and system welfare in terms of system quality of service.

A real-time production scheduling method for RFID-enabled semiconductor back-end shopfloor environment in industry 4.0

PurposeA real-time production scheduling method for semiconductor back-end manufacturing process becomes increasingly important in industry 4.0. Semiconductor back-end manufacturing process is always accompanied by order splitting and merging; besides, in each stage of the process, there are always multiple machine groups that have different production capabilities and capacities. This paper studies a multi-agent based scheduling architecture for the radio frequency identification (RFID)-enabled semiconductor back-end shopfloor, which integrates not only manufacturing resources but also human factors.Design/methodology/approachThe architecture includes a task management (TM) agent, a staff instruction (SI) agent, a task scheduling (TS) agent, an information management center (IMC), machine group (MG) agent and a production monitoring (PM) agent. Then, based on the architecture, the authors developed a scheduling method consisting of capability & capacity planning and machine configuration modules in the TS agent.FindingsThe authors used greedy policy to assign each order to the appropriate machine groups based on the real-time utilization ration of each MG in the capability & capacity (C&C) planning module, and used a partial swarm optimization (PSO) algorithm to schedule each splitting job to the identified machine based on the C&C planning results. At last, we conducted a case study to demonstrate the proposed multi-agent based real-time production scheduling models and methods.Originality/valueThis paper proposes a multi-agent based real-time scheduling framework for semiconductor back-end industry. A C&C planning and a machine configuration algorithm are developed, respectively. The paper provides a feasible solution for semiconductor back-end manufacturing process to realize real-time scheduling.

Multi-Agent Reinforcement Learning Control of a Hydrostatic Wind Turbine-Based Farm

This paper leverages multi-agent reinforcement learning (MARL) to develop an efficient control system for a wind farm comprising a new type of wind turbines with hydrostatic transmission. The primary motivation for hydrostatic wind turbines (HWT) is increased reliability, and reduced manufacturing, operating, and maintaining costs by removing troublesome components and reducing nacelle weight. Nevertheless, the high system complexity of HWT and the wake effect pose significant challenges for the control of HWT-based wind farms. We therefore propose a MARL algorithm named multi-agent policy optimization (MAPO), which allows agents (turbines) to gradually improve their control policies by repeatedly interacting with the environment to learn an optimal operation curve for wind farms. Simulation results based on a wind farm simulator, FAST.Farm, show that MAPO outperforms the greedy policy and a popular learning-based method, multi-agent deep deterministic policy gradient (MADDPG), in terms of power generation.