Reinforcement Algorithm Research Articles

Providing support to operators for monitoring safety functions using reinforcement learning

Continuous monitoring and diagnosis are important for safe operation of nuclear facilities. In an emergency shutdown, the diagnostic tasks can be challenging for human operators who may be under intense stress and/or lack training. In recent years, studies using artificial-intelligence technologies have been actively conducted to help in diagnostic tasks. The current study proposes a data-driven approach that leverages deep reinforcement-learning techniques to intelligently learn effective strategies for state diagnosis of safety functions. First, a learning framework and key elements of reinforcement learning are designed as basic components. Then, a deep neural-network structure and a deep reinforcement algorithm are presented for diagnosis learning. The experimental results demonstrate the feasibility of deep reinforcement learning on diagnosing the safety functions of a nuclear facility.

Transferring decisions to an algorithm: A simple route choice experiment

The future of artificial intelligence is bright and will create new situations in which humans can share or delegate decisions to machines and algorithms – as is likely to be the case in fully autonomous traffic situations. As it is important to understand human-machine interactions and acceptance of these new technologies, we took a deeper look into delegation behavior to an algorithm and its links to personality measures. To this end, we ran the route choice game from Selten et al. (2007), in which 18 players had to choose to either use the main or the side road for 200 periods, but had the opportunity to delegate this decision to a reinforcement algorithm. Results from 216 participants show that delegation rates average around 10% but vary greatly between individual players and with the level of traffic information. Focusing on personality factors, extraverted individuals were robustly found to be more likely to delegate than introvert players. Altogether, results suggest that there is –depending on several factors– hesitation to delegate decisions that affect both the decision maker and other participants.

Reward associations do not explain transitive inference performance in monkeys.

Most accounts of behavior in nonhuman animals assume that they make choices to maximize expected reward value. However, model-free reinforcement learning based on reward associations cannot account for choice behavior in transitive inference paradigms. We manipulated the amount of reward associated with each item of an ordered list, so that maximizing expected reward value was always in conflict with decision rules based on the implicit list order. Under such a schedule, model-free reinforcement algorithms cannot achieve high levels of accuracy, even after extensive training. Monkeys nevertheless learned to make correct rule-based choices. These results show that monkeys' performance in transitive inference paradigms is not driven solely by expected reward and that appropriate inferences are made despite discordant reward incentives. We show that their choices can be explained by an abstract, model-based representation of list order, and we provide a method for inferring the contents of such representations from observed data.

Continuous Reinforcement Algorithm and Robust Economic Dispatching-Based Spot Electricity Market Modeling considering Strategic Behaviors of Wind Power Producers and Other Participants

In a spot wholesale electricity market containing strategic bidding interactions among wind power producers and other participants such as fossil generation companies and distribution companies, the randomly fluctuating natures of wind power hinders not only the modeling and simulating of the dynamic bidding process and equilibrium of the electricity market but also the effectiveness about keeping economy and reliability in market clearing (economic dispatching) corresponding to the independent system operator. Because the gradient descent continuous actor-critic algorithm is demonstrated as an effective method in dealing with Markov’s decision-making problems with continuous state and action spaces and the robust economic dispatch model can optimize the permitted real-time wind power deviation intervals based on wind power producers’ bidding power output, in this paper, considering bidding interactions among wind power producers and other participants, we propose a gradient descent continuous actor-critic algorithm-based hour-ahead electricity market modeling approach with the robust economic dispatch model embedded. Simulations are implemented on the IEEE 30-bus test system, which, to some extent, verifies the market operation economy and the robustness against wind power fluctuations by using our proposed modeling approach.

Deep Reinforcement Learning for Soft, Flexible Robots: Brief Review with Impending Challenges

The increasing trend of studying the innate softness of robotic structures and amalgamating it with the benefits of the extensive developments in the field of embodied intelligence has led to the sprouting of a relatively new yet rewarding sphere of technology in intelligent soft robotics. The fusion of deep reinforcement algorithms with soft bio-inspired structures positively directs to a fruitful prospect of designing completely self-sufficient agents that are capable of learning from observations collected from their environment. For soft robotic structures possessing countless degrees of freedom, it is at times not convenient to formulate mathematical models necessary for training a deep reinforcement learning (DRL) agent. Deploying current imitation learning algorithms on soft robotic systems has provided competent results. This review article posits an overview of various such algorithms along with instances of being applied to real-world scenarios, yielding frontier results. Brief descriptions highlight the various pristine branches of DRL research in soft robotics.

A Neural Network PID-Like Controller Using a Hybrid of Online Actor-Critic Reinforcement Algorithm with the Square Root Cubature Kalman Filter

A Neural Network PID-Like Controller Using a Hybrid of Online Actor-Critic Reinforcement Algorithm with the Square Root Cubature Kalman Filter

Enhancing the efficiency of quantum annealing via reinforcement: A path-integral Monte Carlo simulation of the quantum reinforcement algorithm

The standard quantum annealing algorithm tries to approach the ground state of a classical system by slowly decreasing the hopping rates of a quantum random walk in the configuration space of the problem, where the on-site energies are provided by the classical energy function. In a quantum reinforcement algorithm, the annealing works instead by increasing gradually the strength of the on-site energies according to the probability of finding the walker on each site of the configuration space. Here, by using the path-integral Monte Carlo simulations of the quantum algorithms, we show that annealing via reinforcement can significantly enhance the success probability of the quantum walker. More precisely, we implement a local version of the quantum reinforcement algorithm, where the system wave function is replaced by an approximate wave function using the local expectation values of the system. We use this algorithm to find solutions to a prototypical constraint satisfaction problem (XORSAT) close to the satisfiability to unsatisfiability phase transition. The study is limited to small problem sizes (a few hundreds of variables), nevertheless, the numerical results suggest that quantum reinforcement may provide a useful strategy to deal with other computationally hard problems and larger problem sizes even as a classical optimization algorithm.

LAAPS: an efficient file-based search in unstructured peer-to-peer networks using reinforcement algorithm

An efficient searching algorithm in the absence of control over the object locations is a challenging issue in the design of unstructured peer-to-peer (P2P) networks. In this paper, we report files searching constraints in the unstructured P2P systems. Hence, we exploit learning automata adaptive probabilistic search (LAAPS) algorithm for those constraints which are distributed entirely with adequate bandwidth. LAAPS optimize the learning automata using file container nodes which are leveraged with the types of file. In the simulation, we compare the proposed algorithm with the random probabilistic walk, adaptive search algorithm, and improved adaptive probabilistic search algorithm. The numerical result demonstrates that LAAPS improves the number of hits per query, achieving high success rate with significant message reduction.

Reinforcement learning: The application to autonomous biomimetic underwater vehicles control

The Autonomous Biomimetic Vehicles have been increasing in popularity in the past few years. Controlling such type of vehicles is not trivial: due to its complex dynamics and kinematics, it is complex to analytically derive controllers that can efficiently perform a given task, such as reaching a given position target in a minimum time. In this paper we will evaluate the results of the implementation of a reinforcement algorithm in autonomous biomimetic underwater vehicles, providing a new way to control this type of vehicles in which the algorithm is in constant learning.

Classification Technique of Interviewer-Bot Result using Naïve Bayes and Phrase Reinforcement Algorithms

Students with hectic college schedules tend not to have enough time repeating the course material. Meanwhile, after they graduated, to be accepted in a foreign company with a higher salary, they must be ready for the English-based interview. To meet these needs, they try to practice conversing with someone who is proficient in English. On the other hand, it is not easy to have someone who is not only proficient in English, but also understand about a job interview related topics. This paper presents the development of a machine which is able to provide practice on English-based interviews, specifically on job interviews. Interviewer machine (interviewer bot) is expected to help students practice on speaking English in particular issue of finding suitable job. The interviewer machine design uses words from a chat bot database named ALICE to mimic human intelligence that can be applied to a search engine using AIML. Naïve Bayes algorithm is used to classify the interview results into three categories: POTENTIAL, TALENT and INTEREST students. Furthermore, based on the classification result, the summary is made at the end of the interview session by using phrase reinforcement algorithms. By using this bot, students are expected to practice their listening and speaking skills, also to be familiar with the questions often asked in job interviews so that they can prepare the proper answers. In addition, the bot’ users could know their potential, talent and interest in finding a job, so they could apply to the appropriate companies. Based on the validation results of 50 respondents, the accuracy degree of interviewer chat-bot (interviewer engine) response obtained 86.93%.

Optimization by a quantum reinforcement algorithm

A reinforcement algorithm solves a classical optimization problem by introducing a feedback to the system which slowly changes the energy landscape and converges the algorithm to an optimal solution in the configuration space. Here, we use this strategy to concentrate (localize) preferentially the wave function of a quantum particle, which explores the configuration space of the problem, on an optimal configuration. We examine the method by solving numerically the equations governing the evolution of the system, which are similar to the nonlinear Schr\"odinger equations, for small problem sizes. In particular, we observe that reinforcement increases the minimal energy gap of the system in a quantum annealing algorithm. Our numerical simulations and the latter observation show that such kind of quantum feedbacks might be helpful in solving a computationally hard optimization problem by a quantum reinforcement algorithm.

FLAPS: bandwidth and delay-efficient distributed data searching in Fog-supported P2P content delivery networks

Due to the growing interest for multimedia contents by mobile users, designing bandwidth and delay-efficient distributed algorithms for data searching over wireless (possibly, mobile) “ad hoc” Peer-to-Peer (P2P) content Delivery Networks (CDNs) is a topic of current interest. This is mainly due to the limited computing-plus-communication resources featuring state-of-the-art wireless P2P CDNs. In principle, an effective means to cope with this limitation is to empower traditional P2P CDNs by distributed Fog nodes. Motivated by this consideration, the goal of this paper is twofold. First, we propose and describe the main building blocks of a hybrid (e.g., mixed infrastructure and “ad hoc”) Fog-supported P2P architecture for wireless content delivery, namely, the Fog-Caching P2P architecture. It exploits the topological (possibly, time varying) information locally available at the serving Fog nodes, in order to speed up the data searching operations performed by the served peers. Second, we propose a bandwidth and delay-efficient, distributed and adaptive probabilistic search algorithm, that relies on the learning automata paradigm, e.g., the Fog-supported Learning Automata Adaptive Probabilistic Search (FLAPS) algorithm. The main feature of the FLAPS algorithm is the exploitation of the local topology information provided by the serving Fog nodes and the current status of the collaborating peers, in order to run a suitably distributed reinforcement algorithm for the adaptive discovery of peer-to-peer and peer-to-fog minimum-hop routes. The performance of the proposed FLAPS algorithm is numerically evaluated in terms of Success Rate, Hit-per-Query, Message-per-Query, Response Delay and Message Duplication Factor over a number of randomly generated benchmark CDN topologies. Furthermore, in order to corroborate the actual effectiveness of the FLAPS algorithm, extensive performance comparisons are carried out with some state-of-the-art searching algorithms, namely the Adaptive Probabilistic Search, Improved Adaptive Probabilistic Search and the Random Walk algorithms.

Multichannel speech reinforcement based on binaural unmasking

Speech reinforcement or near-end listening enhancement is a technique that modifies the far-end signal to mitigate the effect of the near-end noise, usually based on the power spectra of the far-end signal and the near-end noise. Psychoacoustic experiments have shown that the location of a noise source with respect to that of a signal source affects the amount of masking. Since conventional speech reinforcement methods obtain spectral gain based only on the power spectra, this psychoacoustic phenomenon called binaural unmasking has not been considered in those approaches. In this paper, we propose a novel speech reinforcement algorithm that modifies the far-end speech signal based on both the power spectrum and the direction-of-arrival (DoA) of the noise. Specifically, we have computed the equivalent frontal noise level from the observed noise level and the estimated DoA, and used it to compute spectral gains as in conventional partial loudness restoration-based speech reinforcement. Experimental results showed that the proposed method outperformed the conventional methods based on partial loudness restoration and speech intelligibility index (SII) optimization in terms of the overall perceived quality through subjective listening tests.

Dynamic spectrum leasing and retail pricing using an experimental economy

This paper presents an agent-based computational model for competitive spectrum leasing and pricing. The model can be used to develop strategies for cognitive operators in regions where there is a heterogeneity of operators due to overlapping coverage areas. We model the interactions among interacting entities using a three-stage agent based model in which we apply the basic principles of an experimental economy. The operators make their decisions regarding the frequency resources to be leased on the wholesale-spectrum market using the Roth-Erev reinforcement algorithm, while on the retail market, they provide services for end-users. Pricing decisions on the retail market are made using a family of pricing strategies, such as linear-reward, successful ratio, and trial-and-error, which are feasible when the bounded rationality of the agents is considered. We show that the profit is significantly affected when forgetting and experimentation parameters are added to reinforce the decisions made on the wholesale market. In more advanced pricing schemes, e.g., successful-ratio and linear-reward, the addition of the forgetting parameter significantly improves the performance in terms of the measured indicators, namely, the average profit and Sharpe ratio. On the other hand, in the zero-intelligence trial-and-error approach, we do not see any positive changes in terms of the measured indicators. In addition, several notable phenomena emerged from the interactions of the agents. For example, the classification of operators who take on the role of spectrum investors as either risk-averse or risk-seeking based on the pricing schemes utilized in the retail market was shown to be an emergent feature of the model.

PbMMD: A novel policy based multi-process malware detection

Contemporary malware makes wide use of techniques to evade popular detection approaches. Behavior-based detection is the most powerful approach to malware detection. This approach is based on system call sequences to model a malicious behavior. A recently immersed malware to defeat behavior-based detection approach is Multi-process malware. This malware is the consequence of multiple processes cooperating to fulfill a malicious task each of which performing a partition of main task and none of them shows an identifiable malicious behavior. In this paper, we have presented a new method called PbMMD for detecting Multi-process malware. In this method, we attempt to inspect the whole processes running on the system and discover collaborative processes by finding processes running along a common execution policy. Beforehand we have learned different execution policy by employing reinforcement algorithm. Finally we decide against a Multi-process malicious behavior by analyzing the cumulative behavior of identified collaborative processes.

New reinforcement algorithms in discontinuous deformation analysis for rock failure

DDARF (Discontinuous Deformation Analysis for Rock Failure) is a numerical algorithm for simulating jointed rock masses' discontinuous deformation. While its reinforcement simulation is only limited to endanchorage bolt, which is assumed to be a linear spring simply. Here, several new reinforcement modes in DDARF are proposed, including lining reinforcement, full-length anchorage bolt and equivalent reinforcement. In the numerical simulation, lining part is assigned higher mechanical strength than surrounding rock masses, it may include multiple virtual joints or not, depending on projects. There must be no embedding or stretching between lining blocks and surrounding blocks. To realize simulation of the full-length anchorage bolt, at every discontinuity passed through the bolt, a set of normal and tangential spring needs to be added along the bolt's axial and tangential direction. Thus, bolt's axial force, shearing force and full-length anchorage effect are all realized synchronously. And, failure criterions of anchorage effect are established for different failure modes. In the meantime, from the perspective of improving surrounding rock masses' overall strength, a new equivalent and tentative simulation method is proposed, it can save calculation storage and improve efficiency. Along the text, simulation algorithms and applications of these new reinforcement modes in DDARF are given.

A Joint Model for Entity Set Expansion and Attribute Extraction from Web Search Queries

Entity Set Expansion (ESE) and Attribute Extraction (AE) are usually treated as two separate tasks in Information Extraction (IE). However, the two tasks are tightly coupled, and each task can benefit significantly from the other by leveraging the inherent relationship between entities and attributes. That is, 1) an attribute is important if it is shared by many typical entities of a class; 2) an entity is typical if it owns many important attributes of a class. Based on this observation, we propose a joint model for ESE and AE, which models the inherent relationship between entities and attributes as a graph. Then a graph reinforcement algorithm is proposed to jointly mine entities and attributes of a specific class. Experimental results demonstrate the superiority of our method for discovering both new entities and new attributes.

Topic Summarization of Microblog Document in Bahasa Indonesia using the Phrase Reinforcement Algorithm

Microblog topic summarization is a part of the challenges to automatically find a topic of any group of microblog posts. This study focused on summarizing Twitter data in Bahasa Indonesia. The main algorithm used in this research is The Phrase Reinforcement Algorithm. This algorithm summarized a group of tweets discussing similar topics using a semi-abstractive approach. As a result of some initial experiments during this study, there are some variations applied in order to obtain summary with a better quality. The evaluation is conducted using human assessment and more than 60% agreed that the summaries have the good grammatical, readability, and informative quality

Random Fourier extreme learning machine with [formula omitted]-norm regularization

This paper proposes a novel algorithm, termed random Fourier extreme learning machine with ℓ2,1-norm regularization, to improve the robustness and compactness of the widely used extreme learning machine. In specific, we firstly introduce the random Fourier mappings as activation functions to approximate the Gaussian kernel, with the aim to improve the extendibility of the powerful kernel ELM algorithms. We then adopt the ℓ2,1-norm to eliminate the potential irrelevant neurons, resulting in a more compact and discriminative hidden layer. After that, we propose an efficient algorithm with proved convergence to solve the resultant optimization problem. Extensive experiments have been conducted on 30 benchmark data sets to compare the proposed algorithm with six popular extreme learning algorithms. As observed, our algorithm outperforms the enumerated hidden layer reinforcement algorithms. In addition, it significantly improves the computational efficiency of Gaussian kernel extreme learning machine with comparable classification and regression performance in large scale learning scenarios.

Twitter Event Summarization Using Phrase Reinforcement Algorithm and NLP Features

Twitter Event Summarization Using Phrase Reinforcement Algorithm and NLP Features