RL Algorithm Research Articles

Deep successor feature learning for text generation

In this paper we present an approach to training neural network to generate sequences using successor feature learning from reinforcement learning. The model can be thought as two components, an MLE-based token generator and an estimator that predicts the future value of whole sentence. As we know, reinforcement learning has been applied to dealing with the exposure bias problem of generating sequences. Compared with other RL algorithm, successor feature(SF) can learn robust value function provided observations and reward by decomposing the value function into two components - a reward predictor and a successor map. The encoder-decoder framework with SF enables the decoder to generate outputs that receive more future reward, which means that the model pays attention on not only the current word but also the rest words. We demonstrate that the approach improves performance on two translation tasks.

Determination of a neutron beam divergence after the rocking curve concept using Richardson–Lucy’s unfolding algorithm

Thermal neutron radiographs acquired under high beam divergences suffer the impact of intense penumbrae degrading their final quality. As a divergence reduction is not always feasible, one possible alternative is an a posteriori treatment to restore the degraded images, such as the Richardson–Lucy — RL unfolding algorithm. Such a procedure requires the characterization of the spoiling agent Point Spread function – PSF in order to apply it in the inverse way. It can be deduced from the blur in the radiograph of a shielding blade edge. This blur depends upon the beam divergence and the object–detector gap. Due to the complex scattering of neutrons along a reactor channel, it is usual to express this divergence as the inverse of an L/D ratio. A novel approach based on the concept of Rocking Curve – RC, a term borrowed from the X-ray diffraction field, has been recently proposed which yielded slightly better quantitative results. After this concept, every sub-element of a surface source emits neutrons anisotropically following a bell-shaped profile. The RC angular semi-width incorporating neutron scattering and geometrical blur, is assigned as the beam divergence. The present work aims at its assessment through a quantitative determination ratified by a qualitative evaluation of radiographs unfolded by the RL algorithm. Its main purpose is an additional ratification of the soundness, consistency and robustness of the RC concept by comparison with formerly obtained quantitative results. In spite of the utterly different approaches and techniques, the outcome has corroborated the novel concept. All data treatment is simple and performed by an ad hoc written Fortran 90 program embedding the required algorithms.

A Deep Hierarchical Reinforcement Learning Algorithm in Partially Observable Markov Decision Processes

In recent years, reinforcement learning has achieved many remarkable successes due to the growing adoption of deep learning techniques and the rapid growth in computing power. Nevertheless, it is well-known that flat reinforcement learning algorithms are often not able to learn well and data-efficient in tasks having hierarchical structures, e.g. consisting of multiple subtasks. Hierarchical reinforcement learning is a principled approach that is able to tackle these challenging tasks. On the other hand, many real-world tasks usually have only partial observability in which state measurements are often imperfect and partially observable. The problems of RL in such settings can be formulated as a partially observable Markov decision process (POMDP). In this paper, we study hierarchical RL in POMDP in which the tasks have only partial observability and possess hierarchical properties. We propose a hierarchical deep reinforcement learning approach for learning in hierarchical POMDP. The deep hierarchical RL algorithm is proposed to apply to both MDP and POMDP learning. We evaluate the proposed algorithm on various challenging hierarchical POMDP.

High quality image-pair-based deblurring method using edge mask and improved residual deconvolution

Image deconvolution problem is a challenging task in the field of image process. Using image pairs could be helpful to provide a better restored image compared with the deblurring method from a single blurred image. In this paper, a high quality image-pair-based deblurring method is presented using the improved RL algorithm and the gain-controlled residual deconvolution technique. The input image pair includes a non-blurred noisy image and a blurred image captured for the same scene. With the estimated blur kernel, an improved RL deblurring method based on edge mask is introduced to obtain the preliminary deblurring result with effective ringing suppression and detail preservation. Then the preliminary deblurring result is served as the basic latent image and the gain-controlled residual deconvolution is utilized to recover the residual image. A saliency weight map is computed as the gain map to further control the ringing effects around the edge areas in the residual deconvolution process. The final deblurring result is obtained by adding the preliminary deblurring result with the recovered residual image. An optical experimental vibration platform is set up to verify the applicability and performance of the proposed algorithm. Experimental results demonstrate that the proposed deblurring framework obtains a superior performance in both subjective and objective assessments and has a wide application in many image deblurring fields.

A localized Richardson-Lucy algorithm for fiber orientation estimation in high angular resolution diffusion imaging.

Diffusion tensor imaging is widely used for studying neural fiber trajectories in white matter and for quantifying changes in tissue using diffusion properties at each voxel in the brain. To better model the nature of crossing fibers within complex architectures, rather than using a simplified tensor model that assumes only a single fiber direction at each image voxel, a model mixing multiple diffusion tensors is used to profile diffusion signals from high angular resolution diffusion imaging (HARDI) data. Based on the HARDI signal and a multiple tensors model, spherical deconvolution methods have been developed to overcome the limitations of the diffusion tensor model when resolving crossing fibers. The Richardson-Lucy algorithm is a popular spherical deconvolution method used in previous work. However, it is based on a Gaussian distribution, while HARDI data are always very noisy, and the distribution of HARDI data follows a Rician distribution. This current work aims to present a novel solution to address these issues. By simultaneously considering both the Rician bias and neighbor correlation in HARDI data, the authors propose a localized Richardson-Lucy (LRL) algorithm to estimate fiber orientations for HARDI data. The proposed method can simultaneously reduce noise and correct the Rician bias. Mean angular error (MAE) between the estimated Fiber orientation distribution (FOD) field and the reference FOD field was computed to examine whether the proposed LRL algorithm offered any advantage over the conventional RL algorithm at various levels of noise. Normalized mean squared error (NMSE) was also computed to measure the similarity between the true FOD field and the estimated FOD filed. For MAE comparisons, the proposed LRL approach obtained the best results in most of the cases at different levels of SNR and b-values. For NMSE comparisons, the proposed LRL approach obtained the best results in most of the cases at b-value = 3000 s/mm(2), which is the recommended schema for HARDI data acquisition. In addition, the FOD fields estimated by the proposed LRL approach in regions of fiber crossing regions using real data sets also showed similar fiber structures which agreed with common acknowledge in these regions. The novel spherical deconvolution method for improved accuracy in investigating crossing fibers can simultaneously reduce noise and correct Rician bias. With the noise smoothed and bias corrected, this algorithm is especially suitable for estimation of fiber orientations in HARDI data. Experimental results using both synthetic and real imaging data demonstrated the success and effectiveness of the proposed LRL algorithm.

Similarity of learned helplessness in human being and fuzzy reinforcement learning algorithms

The sequential and uncontrolled punishments in social life may lead to what psychologists call learned helplessness or depression. Like learning in social life, agents based on Fuzzy Reinforcement Learning FRL sometimes cannot learn well. Experiments show that if an agent continuously performs actions that cause sequential punishments in the beginning of learning, then it does not usually behave well and often selects actions that evoke punishments. Therefore, the learning takes so long or not successful. In this paper, we address this issue called faulty learning in RL algorithm by exploiting learned helplessness. We demonstrate learned helplessness in the training of an agent by FRL algorithm and analyze it. The result of analysis shows that since the action value function is approximated by a fuzzy system; continuous punishments lead all weight parameters of the approximator toward negative amounts. Hence, the agent cannot learn well. To prevent this problem, we propose a new reinforcement function. The proposed reinforcement function is adaptive and depends on the number of visit of the state. Simulation results show that new reinforcement function prevents learned helplessness and improves the learning in terms of learning speed and action quality. The proposed ideas can be used and extended to our social and psychology life.

A Weight-Throwing Clustering Group Mutual Exclusion Algorithm for Mobile Ad Hoc Networks

A mobile ad hoc network can be defined as a network that is spontaneously deployed and is independent of any static network. The network consists of mobiles nodes with wireless interfaces and has arbitrary dynamic topology. The network suffers from frequent link formation and disruption due to the mobility of the nodes. A clustering method is used for obtaining a hierarchical organization for the ad hoc network. In this paper we present a clustering token-based algorithm for group mutual exclusion in ad hoc mobile networks. The proposed algorithm is adapted from the RL algorithm in [1] and utilizes the concept of weight throwing in [27] to solve the group mutual exclusion problem in mobile ad hoc network. The proposed algorithm is sensitive to link forming and link forming. The algorithm ensures the mutual exclusion, the bounded delay and the concurrent entering properties

Imitation and Reinforcement Learning

In this article, we present both novel learning algorithms and experiments using the dynamical system MPs. As such, we describe this MP representation in a way that it is straightforward to reproduce. We review an appropriate imitation learning method, i.e., locally weighted regression, and show how this method can be used both for initializing RL tasks as well as for modifying the start-up phase in a rhythmic task. We also show our current best-suited RL algorithm for this framework, i.e., PoWER. We present two complex motor tasks, i.e., ball-in-a-cup and ball paddling, learned on a real, physical Barrett WAM, using the methods presented in this article. Of particular interest is the ball-paddling application, as it requires a combination of both rhythmic and discrete dynamical systems MPs during the start-up phase to achieve a particular task.

Compulsory Flow Q-Learning: an RL algorithm for robot navigation based on partial-policy and macro-states

Compulsory Flow Q-Learning: an RL algorithm for robot navigation based on partial-policy and macro-states

Compulsory Flow Q-Learning: an RL algorithm for robot navigation based on partial-policy and macro-states

Abstract Reinforcement Learning is carried out on-line, through trial-and-error interactions of the agent with the environment, which can be very time consuming when considering robots. In this paper we contribute a new learning algorithm, CFQLearning, which uses macro-states, a low-resolution discretisation of the state space, and a partial-policy to get around obstacles, both of them based on the complexity of the environment structure. The use of macro-states avoids convergence of algorithms, but can accelerate the learning process. In the other hand, partial-policies can guarantee that an agent fulfils its task, even through macro-state. Experiments show that the CFQ-Learning performs a good balance between policy quality and learning rate.

An error analysis for the calculation of semiintegrals and semiderivatives by the RL algorithm

The numerical approximation of the semiintegral and semiderivative of a function f by the RL algorithm of Oldham and Spanier [1] is examined. An error analysis is given for the case when f has a continuous second derivative. The performance of the algorithm when applied to experimental data is also discussed. Numerical examples are presented to illustrate the theory.