Improve Learning Speed Research Articles

Exploration-Exploitation Strategies in Deep Q-Networks Applied to Route-Finding Problems

Reinforcement learning is a class of algorithm that allows computers to learn how to accumulate rewards effectively in the environment and ultimately get excellent results. Among them, the exploration-exploitation tradeoff is a very important concept, since a good strategy can improve learning speed and final total reward. In this work, we applied DQN algorithm with different exploration-exploitation strategies to solve traditional route-finding problems. The experimental results show that the epsilon greedy strategy with a parabolic drop in epsilon value over reward improvement is the best, while it is not satisfactory after incorporating the softmax function. We hypothesized that the simplicity of the maze we use in this work in which the agent attempts to find the shortest path leads to the inadequacy of applying softmax to further encourage exploration. Future work thus involves experimenting with mazes at different scales and complexities and observing which exploration-exploitation strategies work best in each condition.

Safety-Guaranteed, Accelerated Learning in MDPs with Local Side Information.

In environments with uncertain dynamics, synthesis of optimal control policies mandates exploration. The applicability of classical learning algorithms to real-world problems is often limited by the number of time steps required for learning the environment model. Given some local side information about the differences in transition probabilities of the states, potentially obtained from the agent's onboard sensors, we generalize the idea of indirect sampling for accelerated learning to propose an algorithm that balances between exploration and exploitation. We formalize this idea by introducing the notion of the value of information in the context of a Markov decision process with unknown transition probabilities, as a measure of the expected improvement in the agent's current estimate of transition probabilities by taking a particular action. By exploiting available local side information and maximizing the estimated value of learned information at each time step, we accelerate the learning process and subsequent synthesis of the optimal control policy. Further, we define the notion of agent safety, a vital consideration for physical systems, in the context of our problem. Under certain assumptions, we provide guarantees on the safety of an agent exploring with our algorithm that exploits local side information. We illustrate agent safety and the improvement in learning speed using numerical experiments in the setting of a Mars rover, with data from onboard sensors acting as the local side information.

Study on Improving Learning Speed of Artificial Neural Network Model for Ammunition Stockpile Reliability Classification

Study on Improving Learning Speed of Artificial Neural Network Model for Ammunition Stockpile Reliability Classification

Distributed Deep Reinforcement Learning Method Using Profit Sharing for Learning Acceleration

AbstractProfit Sharing (PS), a reinforcement learning method that strongly reinforces successful experiences, has been shown to contribute to the improvement of learning speed when combined with a deep Q‐network (DQN). We expect a further improvement in learning speed by integrating PS‐based learning and Ape‐X DQN that has state‐of‐the‐art learning speed instead of the DQN. However, PS‐based learning does not use replay memory. In contrast, the Ape‐X DQN requires the use of replay memory because the exploration of the environment for collecting experiences and network training are performed asynchronously. In this study, we propose Learning‐accelerated Ape‐X, which integrates the Ape‐X DQN and PS‐based learning with some improvements including the use of replay memory. We show through numerical experiments that the proposed method improves the scores in Atari 2600 video games in a shorter time than the Ape‐X DQN. © 2020 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.

GENERATING EFFECTIVE INITIATION SETS FOR SUBGOAL-DRIVEN OPTIONS

Options framework is one of the prominent models serving as a basis to improve learning speed by means of temporal abstractions. An option is mainly composed of three elements: initiation set, option’s local policy and termination condition. Although various attempts exist that focus on how to derive high-quality termination conditions for a given problem, the impact of initiation set generation is relatively unexplored. In this work, we propose an effective goal-oriented heuristic method to derive useful initiation set elements via an analysis of the recent history of events. Effectiveness of the method is experimented on various benchmark problems, and the results are discussed.

Fuzzy Removing Redundancy Restricted Boltzmann Machine: improving learning speed and classification accuracy

To improve the feature extraction ability and shorten the learning time, fuzzy removing redundancy restricted Boltzmann machine (F3RBM) is developed. The features extracted by F3RBM with unsupervised learning are imported into support vector machine (SVM) to establish F3RBM-SVM model, which achieves fast and high-precision automatic classification of different samples. To expand the feature extraction capability of restricted Boltzmann machine (RBM), the deterministic parameters of control model are replaced by fuzzy numbers in view of the superiority of fuzzy idea and the redundancy removal mechanism is introduced. Comparing the feature similarity of hidden units with the threshold value, if the similarity is greater than the threshold value, they are considered to be redundant units with the same features. The redundant units are removed to achieve further dimension reduction. Finally, the learning speed, feature extraction ability, and classification accuracy of different models are compared in MINIST handwritten dataset, Fashion MNIST dataset, and Olivetti Face dataset. The experimental results show that the feature extraction capability of FRBM and F3RBM is better than that of RBM. When there are a large number of hidden units, the learning speed of F3RBM is obviously faster than that of FRBM. The features extracted from F3RBM are imported into the SVM to build F3RBM-SVM model, which improves the classification accuracy and learning speed than general classifier. When adding other noises, F3RBM-SVM has better robustness than other models.

HSEM: highly scalable node embedding for link prediction in very large-scale social networks

Very large-scale social networks are typically sparse and dynamic and often have millions of nodes and billions of links. Link prediction in very large-scale networks is a challenging task for most existing methods. This paper investigates the link prediction problem in very large-scale networks. We propose a model, namely, a very large-scale network co-occurrence embedding algorithm (Hsem), which learns the co-occurrence features of node pairs to embed nodes into a vector with a lower and fixed dimension. A damping-based random walk algorithm is also developed to extract the hierarchical structural information of node pairs. Moreover, we design an incremental learning algorithm based on Hsem to meet the requirements of highly dynamic evolution in very large-scale networks, which significantly improves learning speed while maintaining accuracy. Finally, we present controlled experiments employing the proposed and baseline methods on eight challenging real-world large-scale datasets with a fixed dropout percentage and calculate the time overhead and receiver operating characteristics. The results show that Hsem performs comparably to current state-of-the-art methods and consistently outperforms the baselines in different experimental settings.

A fast online multivariable identification method for greenhouse environment control problems

Abstract Growing and pruning radial basis function (GAP-RBF) is extended for identification and control of multivariable nonlinear systems in this work. The proposed MGAP-RBF algorithm utilizes a sliding data window in the growing criterion and limits the number of hidden neurons by introducing a soft constraint in the pruning strategy to reduce the effect of disturbance and to improve learning speed, respectively. The performance of the proposed method is tested through some benchmark problems, and the results show that the proposed method can gain faster speed than the original GAP-RBF method and Ran algorithm, and more importantly, it can obtain an overwhelming advantages especially for some large-scale data sets with some complex attributes. Finally, the proposed method is applied to online PID tuning on a greenhouse environment control process. Simulation results show the proposed MGAP-RBF algorithm has better performance than the traditional RBF method and the original GAP-RBF method, in particular, it is faster and provides a more compact network with reduced computational complexity than the original GAP-RBF method.

Reward shaping for knowledge-based multi-objective multi-agent reinforcement learning

AbstractThe majority of multi-agent reinforcement learning (MARL) implementations aim to optimize systems with respect to a single objective, despite the fact that many real-world problems are inherently multi-objective in nature. Research into multi-objective MARL is still in its infancy, and few studies to date have dealt with the issue of credit assignment. Reward shaping has been proposed as a means to address the credit assignment problem in single-objective MARL, however it has been shown to alter the intended goals of a domain if misused, leading to unintended behaviour. Two popular shaping methods are potential-based reward shaping and difference rewards, and both have been repeatedly shown to improve learning speed and the quality of joint policies learned by agents in single-objective MARL domains. This work discusses the theoretical implications of applying these shaping approaches to cooperative multi-objective MARL problems, and evaluates their efficacy using two benchmark domains. Our results constitute the first empirical evidence that agents using these shaping methodologies can sample true Pareto optimal solutions in cooperative multi-objective stochastic games.

物体識別のためのロボットの行動学習の高速化

物体識別のためのロボットの行動学習の高速化

Structural Plasticity Denoises Responses and Improves Learning Speed.

Despite an abundance of computational models for learning of synaptic weights, there has been relatively little research on structural plasticity, i.e., the creation and elimination of synapses. Especially, it is not clear how structural plasticity works in concert with spike-timing-dependent plasticity (STDP) and what advantages their combination offers. Here we present a fairly large-scale functional model that uses leaky integrate-and-fire neurons, STDP, homeostasis, recurrent connections, and structural plasticity to learn the input encoding, the relation between inputs, and to infer missing inputs. Using this model, we compare the error and the amount of noise in the network's responses with and without structural plasticity and the influence of structural plasticity on the learning speed of the network. Using structural plasticity during learning shows good results for learning the representation of input values, i.e., structural plasticity strongly reduces the noise of the response by preventing spikes with a high error. For inferring missing inputs we see similar results, with responses having less noise if the network was trained using structural plasticity. Additionally, using structural plasticity with pruning significantly decreased the time to learn weights suitable for inference. Presumably, this is due to the clearer signal containing less spikes that misrepresent the desired value. Therefore, this work shows that structural plasticity is not only able to improve upon the performance using STDP without structural plasticity but also speeds up learning. Additionally, it addresses the practical problem of limited resources for connectivity that is not only apparent in the mammalian neocortex but also in computer hardware or neuromorphic (brain-inspired) hardware by efficiently pruning synapses without losing performance.

A cooperative framework of learning automata and its application in tutorial-like system

A novel cooperative framework of learning automata (LA) is presented in this paper. In a system, where different LA can be integrated via the proposed framework, current State of Learning (SoL) index is advocated to evaluate the learning status of individual LA. Based on that learning status, individual LA will adaptively choose an appropriate interaction strategy at cooperative learning phase. Theoretical analysis demonstrated that this index is able to preserve the ε-optimal feature of independent learning automata. Experimental simulations validated that this cooperative framework is effective in improving learning speed of a variety of LA embedded in this framework. Then, we also present an application of the proposed cooperative framework in tutorial-like systems. Compared with existing method, our cooperative framework based method outperforms in both speed and accuracy.

Teaching Robotics at the Primary School: An Innovative Approach

Many researchers and teachers agree that the inclusion of Science, Technology, Engineering, and Math in early education provides a strong motivation and a great improvement in learning speed. Most curricula in primary schools include a number of concepts that cover science and math, but less effort is applied in teaching problem solving, computer science, technology and robotics. The use of robotic systems and the introduction of Robotics as a curricula subject can bring the possibility of transmit to children the basics of technology and to give them other kind of human and organizational values. This work present a new program introduced in an Italian primary school thanks to the collaboration with National Instrument and Università Politecnica delle Marche. The subject of Robotics becomes part of the Primary school curricula for all the five years of formation. The program has allowed the teachers training and a complete way through which children have demonstrated great learning abilities, not only in mere technology but also in collaboration and teamwork.

A Strategy-Aware Technique for Learning Behaviors from Discrete Human Feedback

This paper introduces two novel algorithms for learning behaviors from human-provided rewards. The primary novelty of these algorithms is that instead of treating the feedback as a numeric reward signal, they interpret feedback as a form of discrete communication that depends on both the behavior the trainer is trying to teach and the teaching strategy used by the trainer. For example, some human trainers use a lack of feedback to indicate whether actions are correct or incorrect, and interpreting this lack of feedback accurately can significantly improve learning speed. Results from user studies show that humans use a variety of training strategies in practice and both algorithms can learn a contextual bandit task faster than algorithms that treat the feedback as numeric. Simulated trainers are also employed to evaluate the algorithms in both contextual bandit and sequential decision-making tasks with similar results.

An Intelligent Temperature Controller for the Insulation Tool Storeroom of Power System

The storeroom of live working insulated tools of power system has some special requirements such as temperature and humidity. In general, the temperature is stabilized at a particular value and the humidity is higher than a certain value. The storeroom temperature and humidity are mainly controlled by the heater, dehumidifier and air-conditioning compressor. In order to well maintain the stability of the temperature, fuzzy neural network is used to control temperature which makes parameters and rules optimized by the neural network learning. So the designed controller has strong self-adapting character in controlling process. In order to improve learning speed, GA algorithm is used to optimize those initial values of fuzzy reasoning networks. And in order to overcome the shortcoming of local optimization, designed learning parameters can be adjusted according to error and error change in training. Simulating experiment shows that the designed controller has better controlling effect than other conventional controller and which can be used in the storeroom of live working insulation tools in power system. DOI : http://dx.doi.org/10.11591/telkomnika.v12i3.4448 Full Text: PDF

Learning Transfer Automatic through Data Mining in Reinforcement Learning

One of the problems in reinforcement learning is that as the environment becomes more complex, the number of parameters used in decision making increase which leads us to a slow decision making process. The main idea here is to come up with a new algorithm which is able to transfer the information, using data mining techniques in extracting the patterns. We introduce a new algorithm for state transitions and actions which happen during the transfer by the agent are saved as a data set for data mining techniques which is presented Learning With Action Transfer (LAT). The main idea is to use the repeated action in each state, as a pattern in similar states as a means to improve learning speed and performance. The results in our algorithm will be compared to the results in Qlearning algorithm.. General Terms Reinforcement learning

Data-Efficient Generalization of Robot Skills with Contextual Policy Search

In robotics, controllers make the robot solve a task within a specific context. The context can describe the objectives of the robot or physical properties of the environment and is always specified before task execution. To generalize the controller to multiple contexts, we follow a hierarchical approach for policy learning: A lower-level policy controls the robot for a given context and an upper-level policy generalizes among contexts. Current approaches for learning such upper-level policies are based on model-free policy search, which require an excessive number of interactions of the robot with its environment. More data-efficient policy search approaches are model based but, thus far, without the capability of learning hierarchical policies. We propose a new model-based policy search approach that can also learn contextual upper-level policies. Our approach is based on learning probabilistic forward models for long-term predictions. Using these predictions, we use information-theoretic insights to improve the upper-level policy. Our method achieves a substantial improvement in learning speed compared to existing methods on simulated and real robotic tasks.

Automated Flare Prediction Using Extreme Learning Machine

Extreme learning machine (ELM) is a fast learning algorithm of single-hidden layer feedforward neural networks (SLFNs). Compared with the traditional neural networks, the ELM algorithm has the advantages of fast learning speed and good generalization. At the same time, an ordinal logistic regression (LR) is a statistical method which is conceptually simple and algorithmically fast. In this paper, in order to improve the real-time performance, a flare forecasting method is introduced which is the combination of the LR model and the ELM algorithm. The predictive variables are three photospheric magnetic parameters, that is, the total unsigned magnetic flux, length of the strong-gradient magnetic polarity inversion line, and total magnetic energy dissipation. The LR model is used to map these three magnetic parameters of each active region into four probabilities. Consequently, the ELM is used to map the four probabilities into a binary label which is the final output. The proposed model is used to predict the occurrence of flares with a certain level over 24 hours following the time when the magnetogram is recorded. The experimental results show that the cascade algorithm not only improves learning speed to realize timely prediction but also has higher accuracy of X-class flare prediction in comparison with other methods.

The Design of Image Compression with BP Neural Network Based on the Dynamic Adjusting Hidden Layer Nodes

Aiming at the image compression algorithms with the used BP neural network ,they have inherent defects of poor universality and long training time, a model of the dynamic adjusting hidden layer nodes of BP neural network is designed. According to the training image, using the correlation coefficient and dispersion degree of the same hidden layer’s nodes, we cut and delete some no nodes, this algorithm not only can improve learning speed effectively but also has certain generalization ability, and can complete the task of no- training images compression through experiments.

가변 감쇠 파라미터를 이용한 Levenberg-Marquardt 알고리즘의 학습 속도 향상

Levenberg-Marquardt 알고리즘에서 감쇠 파라미터는 오류역전파 학습과 Gauss-Newton 학습의 스위치 역할을 하며 학습 속도에 영향을 준다. 이런 감쇠 파라미터를 고정시키는 것은 오차 함수의 진동을 유발하고 학습 속도를 감소시킨다. 따라서 본 논문은 오차 함수의 변화 과정을 참조하여 감쇠 파라미터를 가변적으로 적용하는 방법을 제안한다. 제안된 방법은 오차의 변화량이 크면 감쇠 파라미터를 크게, 오차의 변화량이 작으면 감쇠 파라미터를 작게 조정한다. 이것은 모멘텀과 유사한 역할을 하여 학습 속도를 향상시킨다. 제안된 방법의 검증을 위한 실험으로는 iris 분류 문제와 wine 분류 문제를 사용하였다. 제안된 방법은 iris 분류 문제에서는 67% 학습에서, wine 분류 문제에서는 78% 학습에서 학습 속도가 향상되었으며 기존 방법과 비교하여 오차의 진동도 적은 것을 확인할 수 있었다. The damping parameter of Levenberg-Marquardt algorithm switches between error backpropagation and Gauss-Newton learning and affects learning speed. Fixing the damping parameter induces some oscillation of error and decreases learning speed. Therefore, we propose the way of a variable damping parameter with referring to the alternation of error. The proposed method makes the damping parameter increase if error rate is large and makes it decrease if error rate is small. This method so plays the role of momentum that it can improve learning speed. We tested both iris recognition and wine recognition for this paper. We found out that this method improved learning speed in 67% cases on iris recognition and in 78% cases on wine recognition. It was also showed that the oscillation of error by the proposed way was less than those of other algorithms.