Interpretable Automated Feature Engineering: A Comprehensive Review with a Focus on Dynamic and Stationary Environments

The goal of our research is to estimate the quantiles of a distribution from a large set of samples that arrive sequentially. Since the data set is large, the model we choose is that the data cannot be stored, but rather that estimates of the quantiles are computed in a real-time setting. In such settings, classical estimators that require storing the whole history of the data (or stream) cannot be deployed. In this paper, we present an incremental quantile estimator of a distribution, i.e., one that utilizes the previously-computed estimates and only resorts to the last sample for updating these estimates. The state-of-the-art work on obtaining incremental quantile estimators is due to Tierney [12], and is based on the theory of stochastic approximation. However, a primary shortcoming of the latter work is the requirement to incrementally build local approximations of the distribution function in the neighborhood of the quantiles. This requirement, unfortunately, increases the complexity of the algorithm. In addition to treating the case of a constant update parameter, we extend our work to include the case of a decreasing update parameter. Such modification is suitable for the case of a stationary environment where the true quantile is invariant over time. Experimental results demonstrate that our estimator outperforms the state-of-the-art estimators. In addition, it also copes with dynamic environments.

Respiration rate monitoring is beneficial for the diagnosis of a variety of diseases, such as heart failure and sleep disorders. Radio Frequency (RF) based respiration rate monitoring systems, namely ultra-wideband radar and COTS device, have been proposed without requiring any direct contact with the detected person. However, existing RF based systems either require expensive UWB radio (radar based) or work only in stationary environments (COTS device based). To address the limitations of both radar based and COTS device based systems, in this paper, we propose RespiRadio, a system that can detect a person’s respiration rate in dynamic ambient environments via a single TX-RX pair of WiFi cards. The key novelty of RespiRadio is that it overcomes the limit of existing COTS device based respiration rate systems by synthesizing a wider-bandwidth WiFi radio. With the synthesized WiFi radio, we can identify the path reflected by the breathing person and then analyze the periodicity of the signal power measurements only from this path to infer the respiration rate. We experimentally evaluate the performance of RespiRadio in non-static indoor environments and the results demonstrate that the overall estimation error is 0.152 breaths per minute (bpm).

In recent years, the subject of deep reinforcement learning (DRL) has developed very rapidly, and is now applied in various fields, such as decision making and control tasks. However, artificial agents trained with RL algorithms require great amounts of training data, unlike humans that are able to learn new skills from very few examples. The concept of meta-reinforcement learning (meta-RL) has been recently proposed to enable agents to learn similar but new skills from a small amount of experience by leveraging a set of tasks with a shared structure. Due to the task representation learning strategy with few-shot adaptation, most recent work is limited to narrow task distributions and stationary environments, where tasks do not change within episodes. In this work, we address those limitations and introduce a training strategy that is applicable to non-stationary environments, as well as a task representation based on Gaussian mixture models to model clustered task distributions. We evaluate our method on several continuous robotic control benchmarks. Compared with state-of-the-art literature that is only applicable to stationary environments with few-shot adaption, our algorithm first achieves competitive asymptotic performance and superior sample efficiency in stationary environments with zero-shot adaption. Second, our algorithm learns to perform successfully in non-stationary settings as well as a continual learning setting, while learning well-structured task representations. Last, our algorithm learns basic distinct behaviors and well-structured task representations in task distributions with multiple qualitatively distinct tasks.

XCS is widely accepted as one of the most reliable Michigan-style learning classifier system for data mining. Many studies found that XCS is able to provide good generalization using a ternary representation for binary inputs as well as interval representation for continuous-valued inputs. Since distributed data mining is becoming more popular due to massive data sets spread across a network at many organizations, we have proposed an XCS system for distributed data mining called DXCS. DXCS has been tested on binary inputs. The results showed that DXCS does not only achieve as good performance as the centralized XCS system, but also reduces data transmission in the network. In this paper, we further examine DXCS with real-valued inputs in stationary and dynamic environments

Recently, the online matching problem has attracted much attention due to its wide application on real-world decision-making scenarios. In stationary environments, by adopting the stochastic user arrival model, existing methods are proposed to learn dual optimal prices and are shown to achieve a fast regret bound. However, the stochastic model is no longer a proper assumption when the environment is changing, leading to an optimistic method that may suffer poor performance. In this paper, we study the online matching problem in dynamic environments in which the dual optimal prices are allowed to vary over time. We bound the dynamic regret of online matching problem by the sum of two quantities, including a regret of online max-min problem and a dynamic regret of online convex optimization (OCO) problem. Then we propose a novel online approach named Primal-Dual Online Algorithm (PDOA) to minimize both quantities. In particular, PDOA adopts the primal-dual framework by optimizing dual prices with the online gradient descent (OGD) algorithm to eliminate the online max-min problem's regret. Moreover, it maintains a set of OGD experts and combines them via an expert-tracking algorithm, which gives a sublinear dynamic regret bound for the OCO problem. We show that PDOA achieves an O(K sqrt{T(1+P_T)}) dynamic regret where K is the number of resources, T is the number of iterations and P_T is the path-length of any potential dual price sequence that reflects the dynamic environment. Finally, experiments on real applications exhibit the superiority of our approach.

Genetic Algorithms have widely been used for solving optimization problems in stationary environments. In recent years, there has been a growing interest for investigating and improving the performance of these algorithms in dynamic environments where the fitness landscape changes. In this study, we present an extensive comparison of several algorithms with different characteristics on a common platform by using the moving peaks benchmark and by varying problem parameters.

A multi-colony ant colony optimization (ACO) algorithm consists of several colonies of ants. Each colony uses a separate pheromone table in an attempt to maximize the search area explored. Over the years, multi-colony ACO algorithms have been successfully applied on different optimization problems with stationary environments. In this paper, we investigate their performance in dynamic environments. Two types of algorithms are proposed: homogeneous and heterogeneous approaches, where colonies share the same properties and colonies have their own (different) properties, respectively. Experimental results on the dynamic travelling salesman problem show that multi-colony ACO algorithms have promising performance in dynamic environments when compared with single colony ACO algorithms.

Due to the high maneuverability and flexibility, unmanned aerial vehicles (UAVs) have been considered as a promising paradigm to assist mobile edge computing (MEC) in many scenarios including disaster rescue and field operation. Most existing research focuses on the study of trajectory and computation-offloading scheduling for UAV-assisted MEC in stationary environments, and could face challenges in dynamic environments where the locations of UAVs and mobile devices (MDs) vary significantly. Some latest research attempts to develop scheduling policies for dynamic environments by means of reinforcement learning (RL). However, as these need to explore in high-dimensional state and action space, they may fail to cover in large-scale networks where multiple UAVs serve numerous MDs. To address this challenge, we leverage the idea of “divide-and-conquer” and propose HT3O, a scalable scheduling approach for large-scale UAV-assisted MEC. First, HT3O is built with neural networks via deep RL to obtain real-time scheduling policies for MEC in dynamic environments. More importantly, to make HT3O more scalable, we decompose the scheduling problem into two-layered subproblems and optimize them alternately via hierarchical RL. This not only substantially reduces the complexity of each subproblem, but also improves the convergence efficiency. Experimental results show that HT3O can achieve promising performance improvements over state-of-the-art approaches.

Next generation driver assistance systems demand a precise perception of the vicinity of the vehicle. Sensor readings are usually harnessed to gain knowledge of all moving and stationary obstacles. Commonly two paradigms are followed. Stationary environments are well modeled by non-parametric occupancy grids. Contrary, moving objects require a tracking and are not well suited for grid-based techniques. However, tracking objects requires some prior knowledge and parameterization which is inferior for unstructured cluttered obstacles. Herein we augment a grid-based mapping method designed for static environments with object tracking hence complementing both approaches. To this end we classify regions of stereo images into moving and stationary parts. The stationary part is fused in our static grid whereas moving parts are tracked yielding reliable motion estimates. The classifier used to distinguish moving from stationary parts is based on a Sequential Probability Ratio Test (SPRT), a model selection method which blends well into the tracking architecture. Thereby we achieve real-time operability on modest computing hardware.

A survey of swarm intelligence for dynamic optimization: Algorithms and applications

Identifying dependencies among variables in a complex system is an important problem in network science. Structural equation models (SEM) have been used widely in many fields for topology inference, because they are tractable and incorporate exogenous influences in the model. Topology identification based on static SEM is useful in stationary environments; however, in many applications a time-varying underlying topology is sought. This paper presents an online algorithm to track sparse time-varying topologies in dynamic environments and most importantly, performs a detailed analysis on the performance guarantees. The tracking capability is characterized in terms of a bound on the dynamic regret of the proposed algorithm. Numerical tests show that the proposed algorithm can track changes under different models of time-varying topologies.

In this work, we are addressing the problem of cooperative multi-agent learning for distributed decision making in non stationary environments. Our principal focus is to improve learning by exchanging information between local neighbors (agents) and to ensure the adaption to the new environmental form without ignoring knowledge already acquired. First, a distributed dynamic correlation matrix based on multi-Q learning method, presented in [1], is evaluated. To overcome the shortcomings of this method, a new multi-agent reinforcement learning approach and a new cooperative action selection strategy are developed. Several simulation tests are conducted using a cooperative foraging task with a single moving target and show the efficiency of the proposed methods in the case of large, unknown and temporary dynamic environments.

The effect of different representations has been thoroughly analyzed for evolutionary algorithms in stationary environments. However, the role of representations in dynamic environments has been largely neglected so far. In this paper, we empirically compare and analyze three different representations on the basis of a dynamic multi-dimensional knapsack problem. Our results indicate that indirect representations are particularly suitable for the dynamic multi-dimensional knapsack problem, because they implicitly provide a heuristic adaptation mechanism that improves the current solutions after a change.

The vestibulo-ocular reflex (VOR) stabilizes vision during head movements by counter-rotating the eyes in the orbits. Although considered one of the simplest reflexes due to its minimal neuronal circuity comprising a 3-neuron arc, previous studies have shown that VOR performance deteriorates in both monkeys and humans when they are drowsy. Given constant head perturbations under dynamic environments, the VOR has been proposed as a viable biomarker for detecting human drowsiness in automobiles and other moving vehicles. However, under stationary environments where exogenous head movements are absent, its applicability has been questioned. In this study, we demonstrate that each heartbeat generates small yet distinctive head movements, and the VOR compensates for these minor head perturbations. Furthermore, we show that the effectiveness of VOR responses varies with the degree of drowsiness, indicating that the VOR can serve as an indicator of drowsiness, even in stationary contexts such as in classrooms and offices.

Most research in reinforcement learning has focused on stationary environments. In this paper, we propose several adaptations of Q-learning for a dynamic environment, for both single and multiple agents. The environment consists of a grid of random rewards, where every reward is removed after a visit. We focus on experience replay, a technique that receives a lot of attention nowadays, and combine this method with Q-learning. We compare two variations of experience replay, where experiences are reused based on time or based on the obtained reward. For multi-agent reinforcement learning we compare two variations of policy representation. In the first variation the agents share a Q-function, while in the second variation both agents have a separate Q-function. Furthermore, in both variations we test the effect of reward sharing between the agents. This leads to four different multi-agent reinforcement learning algorithms, from which sharing a Q-function and sharing the rewards is the most cooperative method. The results show that in the single-agent environment both experience replay algorithms significantly outperform standard Q-learning and a greedy benchmark agent. In the multi-agent environment the highest maximum reward sum in a trial is achieved by using one Q-function and reward sharing. The highest mean reward sum is obtained with separate Q-functions and separate rewards.