Online Policy Research Articles

Online Algorithms for Optimizing Age of Information in the IoT Systems with Multi-Slot Status Delivery

Age of information (AoI) is recently proposed to measure the freshness of information, which provides a new performance metric for the real-time Internet of Things (IoT). In this letter, we investigate low complexity online algorithms to minimize the average AoI in an IoT system where the nodes are scheduled to sample the status with multiple packets and send it to the destination through noisy channels. Three online policies are proposed: Greedy policy, Max-Ratio policy and Lyapunov drift policy. Simulation results show that our online algorithms outperform the existing suboptimal algorithm and the Lyapunov drift policy yields the best performance.

Sliding mode-based online fault compensation control for modular reconfigurable robots through adaptive dynamic programming

In this paper, a sliding mode (SM)-based online fault compensation control scheme is investigated for modular reconfigurable robots (MRRs) with actuator failures via adaptive dynamic programming. It consists of a SM-based iterative controller, an adaptive robust term and an online fault compensator. For fault-free MRR systems, the SM surface-based Hamilton–Jacobi–Bellman equation is solved by online policy iteration algorithm. The adaptive robust term is added to guarantee the reachable condition of SM surface. For faulty MRR systems, the actuator failure is compensated online to avoid the fault detection and isolation mechanism. The closed-loop MRR system is guaranteed to be asymptotically stable under the developed fault compensation control scheme. Simulation results verify the effectiveness of the present fault compensation control approach.

THE OPPORTUNITY TO ACCESS BETTER INFORMATION TECHNOLOGY

Children are born without being able to choose a number of factors that are attached to them. This is in fact unfair to him in supporting his future. HOI exists to calculate a number of factors that cause imbalance and how much inequality should be reallocated. The usage of cellphone has covered 63.86 percent. The inequality that must be reallocated is 8.76 percent. Per capita income and residence are the dominant influencing factors. In the case of cellphone ownership, the coverage was still 33.12 percent, with an inequality reallocation of 17.05 percent. Per capita income and residence are the dominant factors. Access coverage on new computers reached 17.35 percent with reallocation of inequalities reaching 28.87 percent. Per capita income, residence and certificate of household head are the dominant factors. Meanwhile internet access has covered 35.02 percent with inequality reallocation reaching 17.85 percent. Per capita income, residence and certificate of household head are the most dominant things. An understanding of reality is very important in taking online policies against children that happen to be implemented during the Covid-19 period and possibly afterward.

Education And The Corona Pandemic: Lessons LearnedExploring Teaching And Learning Needs After The Corona Pandemic: Curriculum And Teaching Methods

The education system all over the world is been highly affected by coronavirus and Covid-10 pandemic when the lockdown was announced worldwide. In order to control and prevent the spread of the virus, the schools, colleges, and even the universities was closed. This closure of educational instititutes had bought up number of difficulties for the student, teacher, and the parents as well and inorder to overcome the situation and continue the system of education, online or distance learning policy was offered and adapted by each and everyone that were associated with the educational system. It was a challenge for the developing countries wgwre thre is lack of facilities like internet access, infrastructure, and gadgets as well. There were countries where the fee of internet was reduced, educational resources were provided by the governing bodies so as to fill the gap. There is a huge pressure on the teaching staff to adapt the online teaching mode and conduct the classes by first learning the process for themselves. Everyone had make all the efforts to learn and adapt the new way of teaching and learning and making the process smooth in the coronavirus pandemic situation.

Near-optimal replacement policies for shared caches in multicore processors

An optimal replacement policy that minimizes the miss rate in a private cache was proposed several decades ago. It requires knowing the future access sequence the cache will receive. There is no equivalent for shared caches because replacement decisions alter this future sequence. We present a novel near-optimal policy for minimizing the miss rate in a shared cache that approaches the optimal execution iteratively. During each iteration, the future access sequence is reconstructed on every miss interleaving the future per-core sequences, taken from the previous iteration. This single sequence feeds a classical private-cache optimum replacement policy. Our evaluation on a shared last-level cache shows that our proposal iteratively converges to a near-optimal miss rate that is independent of the initial conditions, within a margin of 0.1%. The best state-of-the-art online policies achieve around 65% of the miss rate reduction obtained by our near-optimal proposal. In a shared cache, miss rate optimization does not imply the optimization of other metrics. Therefore, we also propose a new near-optimal policy to maximize fairness between cores. The best state-of-the-art online policy achieves 60% of the improvement in fairness seen with our near-optimal policy. Our proposals are useful both for setting upper performance bounds and inspiring implementable mechanisms for shared caches.

Caching Policies for Delay Minimization in Small Cell Networks With Coordinated Multi-Point Joint Transmissions

In 5G and beyond network architectures, operators and content providers base their content distribution strategies on Heterogeneous Networks, where macro and small cells are combined to offer better Quality of Service to wireless users. On top of such networks, edge caching and Coordinated Multi-Point (CoMP) joint transmissions are used to further improve performance. In this paper, we address the average delay minimization problem by first formulating it as a static optimization problem. Even though the problem is NP-hard we are able to solve it via an efficient algorithm that guarantees a 1/2-approximation ratio. We then proceed to propose two fully distributed and dynamic caching policies for the same problem. The first one asymptotically converges to the static optimal solution under the Independent Reference Model (IRM). The second one provides better results in practice under real (non-stationary) request processes. Our online policies outperform existing dynamic solutions that are PHY-unaware.

Event‐driven adaptive near‐optimal tracking control of the robot in aircraft skin inspection

AbstractIn this article, we discuss a near‐optimal tracking control problem (NOTCP) of robots used for inspecting aircraft skin with partially unknown systems, unmeasurable states, unknown disturbances, and unknown output delay. A novel observer based on an augmented neural network is designed to overcome the unknown disturbances, unknown output delay, and unknown internal states. An augmented system state, composed of the tracking error and reference system state, is proposed to introduce a new nonquadratic discounted performance function for the NOTCP. Due to the complexity in solving the Hamilton–Jacobi–Bellman equation, an online policy iteration is presented under the adaptive dynamic programming (ADP) framework. Unlike the traditional ADP, the event‐driven algorithm updates the control input only when the event is triggered, which reduces the computational cost and transmission load. Both the control policy and the observer are updated according to the developed triggering condition. Convergence to a near‐optimal control solution and the stability analysis of the proposed algorithm are shown through the Lyapunov candidate function for both the continuous and jump dynamics. The performance of the proposed algorithm is demonstrated by simulation.

Adaptive optimization algorithm for nonlinear Markov jump systems with partial unknown dynamics

AbstractAn online adaptive optimal control problem for a class of nonlinear Markov jump systems (MJSs) is studied. It is worth noting that the dynamic information of MJSs is partially unknown. Applying the neural network linear differential inclusion techniques, the nonlinear terms in MJSs are approximately converted to linear forms. By using subsystem transformation schemes, we can transfer the nonlinear MJSs to N new coupled linear subsystems. Then a new online policy iteration algorithm is put forward to obtain the adaptive optimal controller. Some theorems are given afterward to ensure the convergence of the new algorithm. At last, a simulation example is provided to verify the applicability of the algorithm.

Online Policies for Efficient Volunteer Crowdsourcing

Nonprofit crowdsourcing platforms such as food recovery organizations rely on volunteers to perform time sensitive tasks. Thus, their success crucially depends on efficient volunteer utilization and engagement. To encourage volunteers to complete a task, platforms use nudging mechanisms to notify a subset of volunteers with the hope that at least one of them responds positively. However, since excessive notifications may reduce volunteer engagement, the platform faces a trade-off between notifying more volunteers for the current task and saving them for future ones. Motivated by these applications, we introduce the online volunteer notification problem, a generalization of online stochastic bipartite matching where tasks arrive following a known time-varying distribution over task types. Upon arrival of a task, the platform notifies a subset of volunteers with the objective of minimizing the number of missed tasks. To capture each volunteer’s adverse reaction to excessive notifications, we assume that a notification triggers a random period of inactivity, during which she will ignore all notifications. However, if a volunteer is active and notified, she will perform the task with a given pair-specific match probability that captures her preference for the task. We develop an online randomized policy that achieves a constant-factor guarantee close to the upper bound we establish for the performance of any online policy. Our policy as well as hardness results are parameterized by the minimum discrete hazard rate of the inter-activity time distribution. The design of our policy relies on sparsifying an ex-ante feasible solution by solving a sequence of dynamic programs. Further, in collaboration with Food Rescue U.S., a volunteer-based food recovery platform, we demonstrate the effectiveness of our policy by testing it on the platform's data from various locations across the U.S.

Content Promotion for Online Content Platforms with Network Diffusion Effect

Online content platform aims to maximize content adoption which is simultaneously driven by the platform promotion and network diffusion. We study the candidate generation and promotion (CGP) problem for online content with the diffusion effect in this paper. Motivated by real-world datasets from the industry partner, we propose a novel diffusion model for online content that captures time variant diffusion effect and the existence of promoting probability. Based on the diffusion model, we formulate the CGP problem as a two-stage optimization problem. We show that the problem is NP-hard and can be seen as a submodular maximization problem. By utilizing the two-stage structure and combining the greedy idea from submodularity, we propose an $(1-\frac{1}{e})(1-\epsilon)$-approximation algorithm. We also consider the online version of this problem where the platform is simultaneously estimating the diffusion parameters and deploying the content promotion decisions. We propose a state-dependent non-anticipatory online policy that achieves $\tilde{\mathcal{O}}(\sqrt{T})$ of $\alpha$-Cumulative Myopic Regret ($\alpha$-CMRegret). Numerical results from a real-world dataset show that the diffusion process for online content deviates drastically from the traditional diffusion models and our proposed model provides much better fit. We also demonstrate the efficiency of our offline and online algorithms with the dataset.

Optimal Adversarial Policies in the Multiplicative Learning System With a Malicious Expert

We consider a learning system based on the conventional multiplicative weight (MW) rule that combines experts' advice to predict a sequence of true outcomes. It is assumed that one of the experts is malicious and aims to impose the maximum loss on the system. The system's loss is naturally defined to be the aggregate absolute difference between the sequence of predicted outcomes and the true outcomes. We consider this problem under both offline and online settings. In the offline setting where the malicious expert must choose its entire sequence of decisions a priori, we show somewhat surprisingly that a simple greedy policy of always reporting false prediction is asymptotically optimal with an approximation ratio of 1+O√(ln N)/N, where N is the total number of prediction stages. In particular, we describe a policy that closely resembles the structure of the optimal offline policy. For the online setting where the malicious expert can adaptively make its decisions, we show that the optimal online policy can be efficiently computed by solving a dynamic program in O(N <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> ). We also discuss a generalization of our model to multi-expert settings. Our results provide a new direction for vulnerability assessment of commonly-used learning algorithms to internal adversarial attacks.

VidCloud

As video-streaming services have expanded and improved, cloud-based video has evolved into a necessary feature of any successful business for reaching internal and external audiences. In this article, video streaming over distributed storage is considered where the video segments are encoded using an erasure code for better reliability. We consider a representative system architecture for a realistic (typical) content delivery network (CDN). Given multiple parallel streams/link between each server and the edge router, we need to determine, for each client request, the subset of servers to stream the video, as well as one of the parallel streams from each chosen server. To have this scheduling, this article proposes a two-stage probabilistic scheduling. The selection of video quality is also chosen with a certain probability distribution that is optimized in our algorithm. With these parameters, the playback time of video segments is determined by characterizing the download time of each coded chunk for each video segment. Using the playback times, a bound on the moment generating function of the stall duration is used to bound the mean stall duration. Based on this, we formulate an optimization problem to jointly optimize the convex combination of mean stall duration and average video quality for all requests, where the two-stage probabilistic scheduling, video quality selection, bandwidth split among parallel streams, and auxiliary bound parameters can be chosen. This non-convex problem is solved using an efficient iterative algorithm. Based on the offline version of our proposed algorithm, an online policy is developed where servers selection, quality, bandwidth split, and parallel streams are selected in an online manner. Experimental results show significant improvement in QoE metrics for cloud-based video as compared to the considered baselines.

An Online Policy for Energy-Efficient State Control of Manufacturing Equipment

Machine state control is one of the most promising energy-efficient measures for machining processes. A proper control reduces the energy consumed during idle periods by switching off/on the machines. A critical barrier for practical implementation is related to the knowledge of part arrival process that is affected by uncertainty. The stochastic processes involved in the system are usually assumed to be known. However, real production environments are subject to several sources of randomness that are difficult to model a priori. This work provides an online time-based algorithm that is able to control the machine state. Through a method for the estimation of the stochastic process, the algorithm provides the optimal control parameters based on a collected set of observations. A new policy is formulated to manage the control over time such that changes in the control parameters are applied only under certain conditions. Potential benefits are discussed using realistic numerical cases. Note to Practitioners-This article analyzes the control problem of switching off/on a machine tool for energy saving during machine idle periods. A control policy based on time information is investigated when the machine requires a startup time to resume the service after being switched off. The proposed policy works online while acquiring information from the real system. An algorithm is described for identifying and applying the optimal control parameters. The results of this research will be useful for a practical implementation of a switching policy for energy saving. This implementation requires the estimation of the power adsorbed by the machine in four different states and, therefore, it reduces the implementation effort for practitioners.

AI Driven Heterogeneous MEC System with UAV Assistance for Dynamic Environment: Challenges and Solutions

By taking full advantage of Computing, Communication and Caching (3C) resources at the network edge, Mobile Edge Computing (MEC) is envisioned as one of the key enablers for next generation networks. However, current fixed-lo-cation MEC architecture may not be able to make real-time decision in dynamic environments, especially in large-scale scenarios. To address this issue, in this article, a Heterogeneous MEC (H-MEC) architecture is proposed, which is composed of fixed unit, i.e., Ground Stations (GSs) as well as moving nodes, i.e., Ground Vehicles (GVs) and Unmanned Aerial Vehicles (UAVs), all with 3C resource enabled. The key challenges in H-MEC, i.e., mobile edge node management, real-time decision making, user association and resource allocation along with the possible Artificial Intelligence (AI)-based solutions, are discussed. In addition, the AI-based joint Resource schEduling (ARE) framework with two different AI-based mechanisms, i.e., Deep neural network (DNN)-based and deep reinforcement learning (DRL)-based architectures, are proposed. DNN-based solution with online incremental learning applies the global optimizer and therefore has better performance than the DRL-based architecture with online policy updating, but requires longer training time. The simulation results are given to verify the efficiency of our proposed ARE framework.

Uniform thresholding based transmission policy for energy harvesting wireless sensor nodes in fading channel

The work presents a novel computationally efficient transmission policy for throughput maximization over point-to-point sensor links employing harvest-use-store protocol with finite storage capacity battery. In these settings, under finite averaging duration constraint, the stochastic dynamic programming (SDP) technique provides the optimal solution for throughput maximization, but the implementation complexity for SDP is prohibitively large. Thus, there is a need to explore new solutions that can provide near-optimal throughput with lower implementation complexity. The work in this paper presents a adaptive transmission policy based on uniform thresholding that achieves a near-optimal throughput obtainable by SDP. Quantitative comparison with optimal online policies shows that the proposed policy attains performance close to SDP with lower implementation complexity.

Efficient policy detecting and reusing for non-stationarity in Markov games

One challenging problem in multiagent systems is to cooperate or compete with non-stationary agents that change behavior from time to time. An agent in such a non-stationary environment is usually supposed to be able to quickly detect the other agents’ policy during online interaction, and then adapt its own policy accordingly. This article studies efficient policy detecting and reusing techniques when playing against non-stationary agents in cooperative or competitive Markov games. We propose a new deep Bayesian policy reuse algorithm, a.k.a. DPN-BPR+, by extending the recent BPR+ algorithm with a neural network as the value-function approximator. To detect policy accurately, we propose the rectified belief model taking advantage of the opponent model to infer the other agents’ policy from reward signals and its behavior. Instead of directly storing individual policies as BPR+, we introduce distilled policy network that serves as the policy library, and policy distillation to achieve efficient online policy learning and reuse. DPN-BPR+ inherits all the advantages of BPR+. In experiments, we evaluate DPN-BPR+ in terms of detection accuracy, cumulative reward and speed of convergence in four complex Markov games with raw visual inputs, including two cooperative games and two competitive games. Empirical results show that our proposed DPN-BPR+ approach has better performance than existing algorithms in all these Markov games.

The Bayesian Prophet: A Low-Regret Framework for Online Decision Making

We develop a new framework for designing online policies given access to an oracle providing statistical information about an off-line benchmark. Having access to such prediction oracles enables simple and natural Bayesian selection policies and raises the question as to how these policies perform in different settings. Our work makes two important contributions toward this question: First, we develop a general technique we call compensated coupling, which can be used to derive bounds on the expected regret (i.e., additive loss with respect to a benchmark) for any online policy and off-line benchmark. Second, using this technique, we show that a natural greedy policy, which we call the Bayes selector, has constant expected regret (i.e., independent of the number of arrivals and resource levels) for a large class of problems we refer to as “online allocation with finite types,” which includes widely studied online packing and online matching problems. Our results generalize and simplify several existing results for online packing and online matching and suggest a promising pathway for obtaining oracle-driven policies for other online decision-making settings. This paper was accepted by George Shanthikumar, big data analytics.

Comparing Online Policy Networks: The Cases of Hydraulic Fracturing Regulation in Colorado

Comparing Online Policy Networks: The Cases of Hydraulic Fracturing Regulation in Colorado

Online policy iteration ADP-based attitude-tracking control for hypersonic vehicles

An online adaptive dynamic programming (ADP) attitude-tracking controller based on policy iteration is proposed, aiming to approach the optimal control of hypersonic vehicles (HVs). The Bellman equation, known as the principal recursive dynamic programming formula, is provided to obtain the controller. In particular, the control action is generated by the ADP controller to track the attitude trajectory. In order to approach optimal control in the uncertain nonlinear HVs system, we use policy iteration to approximate the Bellman equation and build an actor-predictor-critic framework, in which the action network, state estimator and critic network are adopted to implement the policy iteration. Meanwhile, an offline learning method is provided to approach the initial value of iterative computations and improve the efficiency of online learning. The comparative simulations demonstrate the good performance of PIADP with aerodynamic parameter perturbations and random disturbances.

Writing Toward Justice: Youth Speculative Civic Literacies in Online Policy Discourse

This article explores how speculative civic literacies can support youth engagement in policy discourse in digital and analog contexts. We broaden the scope of civic literacies by emphasizing core principles of Afrofuturism and participatory culture. Articulating a specific framework for applying these principles to contemporary conceptions of civic literacies, we identify six digital civic literacy practices that may be leveraged in classrooms. We then analyze two case studies of youth engaging with digital tools and web-based platforms, providing examples of how youth participate in educational policy discourse in online contexts.