Pursuit Learning Automata Research Articles

The Hierarchical Discrete Pursuit Learning Automaton: A Novel Scheme With Fast Convergence and Epsilon-Optimality.

Since the early 1960s, the paradigm of learning automata (LA) has experienced abundant interest. Arguably, it has also served as the foundation for the phenomenon and field of reinforcement learning (RL). Over the decades, new concepts and fundamental principles have been introduced to increase the LA's speed and accuracy. These include using probability updating functions, discretizing the probability space, and using the "Pursuit" concept. Very recently, the concept of incorporating "structure" into the ordering of the LA's actions has improved both the speed and accuracy of the corresponding hierarchical machines, when the number of actions is large. This has led to the ϵ -optimal hierarchical continuous pursuit LA (HCPA). This article pioneers the inclusion of all the above-mentioned phenomena into a new single LA, leading to the novel hierarchical discretized pursuit LA (HDPA). Indeed, although the previously proposed HCPA is powerful, its speed has an impediment when any action probability is close to unity, because the updates of the components of the probability vector are correspondingly smaller when any action probability becomes closer to unity. We propose here, the novel HDPA, where we infuse the phenomenon of discretization into the action probability vector's updating functionality, and which is invoked recursively at every stage of the machine's hierarchical structure. This discretized functionality does not possess the same impediment, because discretization prohibits it. We demonstrate the HDPA's robustness and validity by formally proving the ϵ -optimality by utilizing the moderation property. We also invoke the submartingale characteristic at every level, to prove that the action probability of the optimal action converges to unity as time goes to infinity. Apart from the new machine being ϵ -optimal, the numerical results demonstrate that the number of iterations required for convergence is significantly reduced for the HDPA, when compared to the state-of-the-art HCPA scheme.

Adaptive pursuit learning for energy‐efficient target coverage in wireless sensor networks

SummaryWith the proliferation of technologies such as wireless sensor networks (WSNs) and the Internet of things (IoT), we are moving towards the era of automation without any human intervention. Sensors are the principal components of the WSNs that bring the idea of IoT into reality. Over the last decade, WSNs are being used in many application fields such as target coverage, battlefield surveillance, home security, health care monitoring, and so on. However, the energy efficiency of the sensor nodes in WSN remains a challenging issue due to the use of a small battery. Moreover, replacing the batteries of the sensor nodes deployed in a hostile environment frequently is not a feasible option. Therefore, intelligent scheduling of the sensor nodes for optimizing its energy‐efficient operation and thereby extending the life‐time of WSN has received a lot of research attention lately. In particular, this article investigates extending the lifetime of the WSN in the context of target coverage problems. To tackle this problem, we propose a scheduling technique for WSN based on a novel concept within the theory of learning automata (LA) called pursuit LA. Each sensor node in the WSN is equipped with an LA so that it can autonomously select its proper state, that is, either sleep or active, with an aim to cover all targets with the lowest energy cost possible. Our comprehensive experimental testing of the proposed algorithm not only verifies the efficiency of our algorithm, but it also demonstrates its ability to yield a near‐optimal solution. The results are promising, given the low computational footprint of the algorithm.

A team of pursuit learning automata for solving deterministic optimization problems

Learning Automata (LA) is a popular decision-making mechanism to “determine the optimal action out of a set of allowable actions” [1]. The distinguishing characteristic of automata-based learning is that the search for an optimal parameter (or decision) is conducted in the space of probability distributions defined over the parameter space, rather than in the parameter space itself [2]. In this paper, we propose a novel LA paradigm that can solve a large class of deterministic optimization problems. Although many LA algorithms have been devised in the literature, those LA schemes are not able to solve deterministic optimization problems as they suppose that the environment is stochastic. In this paper, our proposed scheme can be seen as the counterpart of the family of pursuit LA developed for stochastic environments [3]. While classical pursuit LAs can pursue the action with the highest reward estimate, our pursuit LA rather pursues the collection of actions that yield the highest performance by invoking a team of LA. The theoretical analysis of the pursuit scheme does not follow classical LA proofs, and can pave the way towards more schemes where LA can be applied to solve deterministic optimization problems. Furthermore, we analyze the scheme under both a constant learning parameter and a time-decaying learning parameter. We provide some experimental results that show how our Pursuit-LA scheme can be used to solve the Maximum Satisfiability (Max-SAT) problem. To avoid premature convergence and better explore the search space, we enhance our scheme with the concept of artificial barriers recently introduced in [4]. Interestingly, although our scheme is simple by design, we observe that it performs well compared to sophisticated state-of-the-art approaches.

Distributed learning automata-based scheme for classification using novel pursuit scheme

Learning Automata (LA) is a popular decision making mechanism to “determine the optimal action out of a set of allowable actions” (Agache and Oommen, IEEE Trans Syst Man Cybern-Part B Cybern 2002(6): 738–749, 2002). The distinguishing characteristic of automata-based learning is that the search for the optimising parameter vector is conducted in the space of probability distributions defined over the parameter space, rather than in the parameter space itself (Thathachar and Sastry, IEEE Trans Syst Man Cybern-Part B Cybern 32(6): 711–722, 2002). Recently, Goodwin and Yazidi pioneered the use of Ant Colony Optimisation (ACO) for solving classification problems (Goodwin and Yazidi 2016). In this paper, we propose a novel classifier based on the theory of LA. The classification problem is formulated as a deterministic optimization problem involving a team of LA that operate collectively to optimize an objective function. Although many LA algorithms have been devised in the literature, those LA schemes are not able to solve deterministic optimization problems as they suppose that the environment is stochastic. In this paper, we develop a novel pursuit LA which can be seen as the counterpart of the family of pursuit LA developed for stochastic environments (Agache and Oommen, IEEE Trans Syst Man Cybern Part B Cybern 32(6): 738–749, 2002). While classical pursuit LA are able to pursue the action with the highest reward estimate, our pursuit LA rather pursues the collection of actions that yield the highest performance. The theoretical analysis of the pursuit scheme does not follow classical LA proofs and can pave the way towards more schemes where LA can be applied to solve deterministic optimization problems. When applied to classification, the essence of our scheme is to search for a separator in the feature space by imposing a LA based random walk in a grid system. To each node in the gird we attach an LA, whose actions are the choice of the edges forming the separator. The walk is self-enclosing, i.e., a new random walk is started whenever the walker returns to starting node forming a closed classification path yielding a multiedged polygon. In our approach, the different LA attached at the different nodes search for a polygon that best encircles and separates each class. Based on the obtained polygons, we perform classification by labelling items encircled by a polygon as part of a class using ray casting function. Seen from a methodological perspective, PolyPursuit-LA has appealing properties compared to SVM. In fact, unlike PolyPursuit-LA, the SVM performance is dependent on the right choice of kernel function (e.g. Linear Kernel, Gaussian Kernel)— which is considered a “black art”. PolyPursuit-LA can find arbitrarily complex separators in the feature space. Experimental results from both synthetic and real-life data show that our scheme is able to perfectly separate both simple and complex patterns outperforming existing classifiers, including polynomial and linear SVM, without the need of any “kernel trick”. We believe that the results are impressive given the simplicity of PolyPursuit-LA compared to other approaches such as SVM.

The Hierarchical Continuous Pursuit Learning Automation: A Novel Scheme for Environments With Large Numbers of Actions.

Although the field of learning automata (LA) has made significant progress in the past four decades, the LA-based methods to tackle problems involving environments with a large number of actions is, in reality, relatively unresolved. The extension of the traditional LA to problems within this domain cannot be easily established when the number of actions is very large. This is because the dimensionality of the action probability vector is correspondingly large, and so, most components of the vector will soon have values that are smaller than the machine accuracy permits, implying that they will never be chosen. This paper presents a solution that extends the continuous pursuit paradigm to such large-actioned problem domains. The beauty of the solution is that it is hierarchical, where all the actions offered by the environment reside as leaves of the hierarchy. Furthermore, at every level, we merely require a two-action LA that automatically resolves the problem of dealing with arbitrarily small action probabilities. In addition, since all the LA invoke the pursuit paradigm, the best action at every level trickles up toward the root. Thus, by invoking the property of the "max" operator, in which the maximum of numerous maxima is the overall maximum, the hierarchy of LA converges to the optimal action. This paper describes the scheme and formally proves its ϵ -optimal convergence. The results presented here can, rather trivially, be extended for the families of discretized and Bayesian pursuit LA too. This paper also reports extensive experimental results (including for environments having 128 and 256 actions) that demonstrate the power of the scheme and its computational advantages. As far as we know, there are no comparable pursuit-based results in the field of LA. In some cases, the hierarchical continuous pursuit automaton requires less than 18% of the number of iterations than the benchmark LR-I scheme, which is, by all metrics, phenomenal.

A new methodology for identifying unreliable sensors in data fusion

Sensor fusion is a fundamental research topic that has received significant attention in the literature. An important body of research has focused on assessing the reliability of a sensor or more generally an “information source” by comparing the readings with the ground truth in an online or offline manner. The Weighted Majority Voting algorithm [25], a well-known online learning algorithm, is a typical example of a class of approaches that assess the reliability of a sensor by comparing its readings to the ground truth in a online manner. Unlike the latter stream of research, in this article, we tackle the problem of identifying unreliable sensors without the knowledge of the ground truth– which is a novel research direction in its own right. We advocate that comparing the readings of a sensor to the rest of the sensors gives an invaluable information about its reliability. In this article, we present a solution to the problem based on the theory of S-Model Learning Automata (LA) [17]. Interestingly, the feedback to the S-Model environment LA is defined in an intuitive manner, namely, it is proportional to the number of sensors adhering to the chosen action. Our solution does not impose any constraint on the parity of the number of sensors and thus is general and can handle any arbitrary number of sensors. Apart from applying the classical S-Model LA, we develop a novel S-Model based pursuit LA algorithm that achieves a faster convergence than the legacy solution by an order of magnitude of ten while still yielding high accuracy. The devised schemes have been subjected to comprehensive experiments including comparison to the state-of-the-art.

A novel parallel framework for pursuit learning schemes

Parallel operation of learning automata (LA), which is proposed by Thathachar and Arvind, is a promising mechanism that can reduce the computational burden without compromising accuracy. However, as far as we know, this parallel mechanism has not been widely used due to two reasons: one is the fact that the environment can response to multi-actions simultaneously are few, the other is the relatively slow speed of the learning process.In this paper, a novel parallel framework is presented to reduce the number of required interactions between the incorporated pursuit LA and the environment by introducing decentralized learning and centralized fusion. The philosophy is to learn various aspects of the problem at hand by taking advantage of the diverse exploration of decentralized learning and summarize the common knowledge learned by centralized fusion. Simulations are conducted to verify the effectiveness of our framework and demonstrate its outperforming. The proposed framework is further applied to the stochastic point location problem and obtains an attractive performance.

Fast and Epsilon-Optimal Discretized Pursuit Learning Automata.

Learning automata (LA) are powerful tools for reinforcement learning. A discretized pursuit LA is the most popular one among them. During an iteration its operation consists of three basic phases: 1) selecting the next action; 2) finding the optimal estimated action; and 3) updating the state probability. However, when the number of actions is large, the learning becomes extremely slow because there are too many updates to be made at each iteration. The increased updates are mostly from phases 1 and 3. A new fast discretized pursuit LA with assured ε -optimality is proposed to perform both phases 1 and 3 with the computational complexity independent of the number of actions. Apart from its low computational complexity, it achieves faster convergence speed than the classical one when operating in stationary environments. This paper can promote the applications of LA toward the large-scale-action oriented area that requires efficient reinforcement learning tools with assured ε -optimality, fast convergence speed, and low computational complexity for each iteration.

On incorporating the paradigms of discretization and Bayesian estimation to create a new family of pursuit learning automata

There are currently two fundamental paradigms that have been used to enhance the convergence speed of Learning Automata (LA). The first involves the concept of utilizing the estimates of the reward probabilities, while the second involves discretizing the probability space in which the LA operates. This paper demonstrates how both of these can be simultaneously utilized, and in particular, by using the family of Bayesian estimates that have been proven to have distinct advantages over their maximum likelihood counterparts. The success of LA-based estimator algorithms over the classical, Linear Reward-Inaction (L RI )-like schemes, can be explained by their ability to pursue the actions with the highest reward probability estimates. Without access to reward probability estimates, it makes sense for schemes like the L RI to first make large exploring steps, and then to gradually turn exploration into exploitation by making progressively smaller learning steps. However, this behavior becomes counter-intuitive when pursuing actions based on their estimated reward probabilities. Learning should then ideally proceed in progressively larger steps, as the reward probability estimates turn more accurate. This paper introduces a new estimator algorithm, the Discretized Bayesian Pursuit Algorithm (DBPA), that achieves this by incorporating both the above paradigms. The DBPA is implemented by linearly discretizing the action probability space of the Bayesian Pursuit Algorithm (BPA) (Zhang et al. in IEA-AIE 2011, Springer, New York, pp. 608–620, 2011). The key innovation of this paper is that the linear discrete updating rules mitigate the counter-intuitive behavior of the corresponding linear continuous updating rules, by augmenting them with the reward probability estimates. Extensive experimental results show the superiority of DBPA over previous estimator algorithms. Indeed, the DBPA is probably the fastest reported LA to date. Apart from the rigorous experimental demonstration of the strength of the DBPA, the paper also briefly records the proofs of why the BPA and the DBPA are ϵ-optimal in stationary environments.

Dynamic Channel Allocation in Wireless Networks Using Adaptive Learning Automata

Single-channel based wireless networks have limited bandwidth and throughput and the bandwidth utilization decreases with increased number of users. To mitigate this problem, simultaneous transmission on multiple channels is considered as an option. In this paper, we propose a distributed dynamic channel allocation scheme using adaptive learning automata for wireless networks whose nodes are equipped with single-radio interfaces. The proposed scheme, Adaptive Pursuit learning automata runs periodically on the nodes, and adaptively finds the suitable channel allocation in order to attain a desired performance. A novel performance index, which takes into account the throughput and the energy consumption, is considered. The proposed learning scheme adapts the probabilities of selecting each channel as a function of the error in the performance index at each step. The extensive simulation results in static and mobile environments provide that the proposed channel allocation schemes in the multiple channel wireless networks significantly improves the throughput, drop rate, energy consumption per packet and fairness index—compared to the 802.11 single-channel, and 802.11 with randomly allocated multiple channels. Also, it was demonstrated that the Adaptive Pursuit Reward-Only (PRO) scheme guarantees updating the probability of the channel selection for all the links—even the links whose current channel allocations do not provide a satisfactory performance—thereby reducing the frequent channel switching of the links that cannot achieve the desired performance.

Continuous and discretized pursuit learning schemes: various algorithms and their comparison

A learning automaton (LA) is an automaton that interacts with a random environment, having as its goal the task of learning the optimal action based on its acquired experience. Many learning automata (LAs) have been proposed, with the class of estimator algorithms being among the fastest ones, Thathachar and Sastry, through the pursuit algorithm, introduced the concept of learning algorithms that pursue the current optimal action, following a reward-penalty learning philosophy. Later, Oommen and Lanctot extended the pursuit algorithm into the discretized world by presenting the discretized pursuit algorithm, based on a reward-inaction learning philosophy. In this paper we argue that the reward-penalty and reward-inaction learning paradigms in conjunction with the continuous and discrete models of computation, lead to four versions of pursuit learning automata. We contend that a scheme that merges the pursuit concept with the most recent response of the environment, permits the algorithm to utilize the LAs long-term and short-term perspectives of the environment. In this paper, we present all four resultant pursuit algorithms, prove the E-optimality of the newly introduced algorithms, and present a quantitative comparison between them.

Discretized pursuit learning automata

The problem of a stochastic learning automaton interacting with an unknown random environment is considered. The fundamental problem is that of learning, through interaction, the best action allowed by the environment (i.e. the action that is rewarded optimally). By using running estimates of reward probabilities to learn the optimal action, an extremely efficient pursuit algorithm (PA), which is presently among the fastest algorithms known, was reported in earlier works. The improvements gained by rendering the PA discrete are investigated. This is done by restricting the probability of selecting an action to a finite and, hence, discrete subset of (0, 1). This improved scheme is proven to be epsilon -optimal in all stationary environments. Furthermore, the experimental results seem to indicate that the algorithm presented is faster than the fastest nonestimator learning automata reported to date, and also faster than the continuous pursuit automaton. >