High-dimensional Observation Space Research Articles

Generalization Enhancement of Visual Reinforcement Learning through Internal States.

Visual reinforcement learning is important in various practical applications, such as video games, robotic manipulation, and autonomous navigation. However, a major challenge in visual reinforcement learning is the generalization to unseen environments, that is, how agents manage environments with previously unseen backgrounds. This issue is triggered mainly by the high unpredictability inherent in high-dimensional observation space. To deal with this problem, techniques including domain randomization and data augmentation have been explored; nevertheless, these methods still cannot attain a satisfactory result. This paper proposes a new method named Internal States Simulation Auxiliary (ISSA), which uses internal states to improve generalization in visual reinforcement learning tasks. Our method contains two agents, a teacher agent and a student agent: the teacher agent has the ability to directly access the environment's internal states and is used to facilitate the student agent's training; the student agent receives initial guidance from the teacher agent and subsequently continues to learn independently. From another perspective, our method can be divided into two phases, the transfer learning phase and traditional visual reinforcement learning phase. In the first phase, the teacher agent interacts with environments and imparts knowledge to the vision-based student agent. With the guidance of the teacher agent, the student agent is able to discover more effective visual representations that address the high unpredictability of high-dimensional observation space. In the next phase, the student agent autonomously learns from the visual information in the environment, and ultimately, it becomes a vision-based reinforcement learning agent with enhanced generalization. The effectiveness of our method is evaluated using the DMControl Generalization Benchmark and the DrawerWorld with texture distortions. Preliminary results indicate that our method significantly improves generalization ability and performance in complex continuous control tasks.

Combined Reinforcement Learning and CPG Algorithm to Generate Terrain-Adaptive Gait of Hexapod Robots

Terrain adaptation research can significantly improve the motion performance of hexapod robots. In this paper, we propose a method that combines reinforcement learning with a central pattern generator (CPG) to enhance the terrain adaptation of hexapod robots in terms of gait planning. The hexapod robot’s complex task presents a high-dimensional observation and action space, which makes it challenging to directly apply reinforcement learning to robot control. Therefore, we utilize the CPG algorithm to generate the rhythmic gait while compressing the action space dimension of the agent. Additionally, the proposed method requires less internal sensor information, which exhibits strong applicability. Finally, we conduct experiments and deploy the proposed framework in the simulation environment. The results show that the terrain adaptation policy trained in our framework enables the hexapod robot to move more smoothly and efficiently on rugged terrain compared to the traditional CPG method.

A Novel Outlier Detection Model for Vibration Signals Using Transformer Networks

Outlier detection in vibration signals can play an important role in addressing the issue of structural or environmental changes during vibration testing. In this study, a transformer-based model for outlier detection is proposed. Unlike previous statistical and regression outlier detection methods, the proposed model can identify the outlier location in a high dimensional observation space using the self-attention mechanism. The location of outliers within the vibration observation is marked by a combination of a spatial label and a temporal label. The outlier detection performance of the model is verified by a numerical study of the plane wave and an experimental study of the vibrating plate. These two studies show that the proposed model has good label prediction accuracies(all above 85%) toward the outlier location within the plane wave and vibrating plate observations.

Using Deep Deterministic Policy Gradient (DDPG) to Train a Double-Jointed Arm to Reach Target Locations in the Unity ML-Agents Reacher Environment

The Q-learning process including some of its variants such as Deep Q-Network (DQN) uses an indirect approach to train reinforcement learning agents. Generally, we calculate the Q-values for state-action pairs and then take the maximum or highest value from the list of actions possible in an environment. This is an indirect approach, perhaps inefficient or intractable for training agents with continuous action space. Put another way, Q-learning based methods are only ideal for problems with discrete action spaces. (Google DeepMind/Lillicrap et al., 2016) in their paper entitled “Continuous Control with Deep Reinforcement Learning” in fact argued that “while DQN solves problems with high-dimensional observation spaces, it can only handle discrete and low-dimensional action spaces”. They stated further that “DQN cannot be straightforwardly applied to continuous domains since it relies on finding the action that maximizes the action-value function, which in the continuous-valued case requires an iterative optimization process at every step”. But what if we could bypass calculating Q-values and directly learn a policy from the environment? That is, instead of training our system to generate action values, we instead train it to generate the probability of selecting the right actions? It turns out that this paradigm of training agents actually exists. (Zai & Brown 2020) stated that “this class of algorithms is called policy gradient methods”.

Automatic optic disc detection using low-rank representation based semi-supervised extreme learning machine

Optic disc detection plays an important role in developing automatic screening systems for diabetic retinopathy. Several supervised learning-based approaches have been proposed for optic disc detection. However, these approaches demand that the input training examples are completely labelled. Essentially, in medical image analysis, it is difficult to prepare several training samples which were given reliable class labels due to the fact that manually labelling data is very expensive. Moreover, retinal images such as complex vessels structures in the optic disc constituting nonlinear relationships in high-dimensional observation space, which cannot work well by traditional linear classifiers. In this study, a novel approach named low-rank representation based semi-supervised extreme learning machine (LRR-SSELM) is proposed for automated optic disc detection. Our model has the following advantages. First, it detects the optic disc from the viewpoint of semi-supervised learning and overcomes the problem there are small portion of labelled samples. Second, a nonlinear classifier is introduced into our model to fully explore the nonlinear data. Third, the local and global structures of original data can be greatly persevered by low-rank representation (LRR). The performance of the proposed method is validated on three publicly available databases, DIARETDB0, DIARETDB1 and Messidor. The experimental results indicate the advantages and effectiveness of the proposed approach.

Enhanced Probabilistic Inference Algorithm Using Probabilistic Neural Networks for Learning Control

In model-based methods of reinforcement learning (RL), the probabilistic inference for learning control (PILCO) algorithm, which relies on Gaussian process (GP) for building probabilistic dynamics models, is getting attention for its advantages of requiring only a small amount of data, and being able to learn from scratch in a few trials by explicitly incorporating model uncertainty into long-term planning. However, the time complexity of the GP, which is cubic with respect to the number of trials, makes it challenging to scale the framework to high-dimensional observation spaces. Moreover, the cost function of a task is limited as a locally quadratic function for calculating the policy gradient analytically. To solve these problems, we proposed a probabilistic neural network (PNN) to replace the GP in building probabilistic dynamics models and develop a deterministic control policy by using long-term predictions. In particular, we determine model uncertainty through basic prior knowledge and calculate cumulative cost by sampling from state distributions. This approach can help reduce the influence of model error and time consumption. Compared with the state-of-the-art model-based RL, the proposed approach can reconcile data efficiency and speed of learning even in high-dimensional observation spaces.

Multi-activity 3D human motion recognition and tracking in composite motion model with synthesized transition bridges

Recognizing and tracking multiple activities are all extremely challenging machine vision tasks due to diverse motion types included and high-dimensional (HD) state space. To overcome these difficulties, a novel generative model called composite motion model (CMM) is proposed. This model contains a set of independent, low-dimensional (LD), and activity-specific manifold models that effectively constrain the state search space for 3D human motion recognition and tracking. This separate modeling of activity-specific movements can not only allow each manifold model to be optimized in accordance with only its respective movement, but also improve the scalability of the models. For accurate tracking with our CMM, a particle filter (PF) method is thus employed and then the particles can be distributed in all manifold models at each time step. In addition, an efficient activity switching strategy is proposed to dominate the particle distribution on all LD manifolds. To diffuse the particles amongst manifold models and respond quickly to the sudden changes in the activity, a set of visually-reasonable and kinematically-realistic transition bridges are synthesized by using the good properties of LD latent space and HD observation space, which enables the inter-activity motions seem more natural and realistic. Finally, a pose hypothesis that can best interpret the visual observation is selected and then used to recognize the activity that is currently observed. Extensive experiments, via qualitative and quantitative analyses, verify the effectiveness and robustness of our proposed CMM in the tasks of multi-activity 3D human motion recognition and tracking.

A Structured Sparse Plus Structured Low-Rank Framework for Subspace Clustering and Completion

Recent advances in a low-rank matrix completion have enabled the exact recovery of incomplete data drawn from a low-dimensional subspace of a high-dimensional observation space. However, in many applications, the data are drawn from multiple low-dimensional subspaces without knowing which point belongs to which subspace. In such cases, using a single low-dimensional subspace to complete the data may lead to erroneous results, because the complete data matrix need not be low rank. In this paper, we propose a structured sparse plus structured low-rank ( $\text{S}^3$ LR) optimization framework for clustering and completing data drawn from a union of low-dimensional subspaces. The proposed $\text{S}^3$ LR framework exploits the fact that each point in a union of subspaces can be expressed as a sparse linear combination of all other points and that the matrix of the points within each subspace is low rank . This framework leads to a nonconvex optimization problem, which we solve efficiently by using a combination of a linearized alternating direction method of multipliers and spectral clustering. In addition, we discuss the conditions that guarantee the exact matrix completion in a union of subspaces. Experiments on synthetic data, motion segmentation data, and cancer gene data validate the effectiveness of the proposed approach.

Dimensionality Reduction and Generation of Human Motion

To reuse existing motion data and generate new motion, a method of human motion nonlinear dimensionality reduction and generation, based on fast adaptive scaled Gaussian process latent variable models, is proposed. Through statistical learning on motion data, the motion data are mapped from high-dimensional observation space to low-dimensional latent space to implement nonlinear dimensionality reduction, and probability distributing of posture space which measures the nature of posture is obtained. The posture which meets constraints and has maximal probability can be computed as the solution of inverse kinematics. This method can avoid cockamamie computation and posture distortion existing in traditional inverse kinematics.The experiments show that our method has higher convergence velocity and precision and extends editing range of motion by adapting motion editing direction.

Learning the shape manifold to improve object recognition

This paper presents an approach for object recognition and shape retrieval in binary images. Object description is accomplished by contour and region-based solutions. Fusion of two categories of shape descriptors causes a considerable performance improvement in retrieval. In order to improve the recognition rate, it is important to extract the intrinsic object feature vectors in such a way that the geometrical distance between two such vectors matches the semantic relation between the two objects from which they were extracted. The main contribution in this paper is learning how to map the samples in high-dimensional observation space into the new manifold space so that the geometrically closer vectors belong to near semantics. Feature extraction is done by Isomap that belongs to a non-linear feature extraction algorithm. In the proposed method, the shortest path in the dissimilarity graph leads to less value between the samples in the same category by denoising the edge values. Experiments show that the geometrical distance between the samples on the manifold space are more compatible to the semantic distance of them. The proposed method has been compared to some well-known approaches by a variety of shape databases which includes MPEG-7 Part B, Kimia silhouettes, and Fish. The results show the effectiveness and validity of the method.

Guided Locally Linear Embedding

Nonlinear dimensionality reduction is the problem of retrieving a low-dimensional representation of a manifold that is embedded in a high-dimensional observation space. Locally Linear Embedding (LLE), a prominent dimensionality reduction technique is an unsupervised algorithm; as such, it is not possible to guide it toward modes of variability that may be of particular interest. This paper proposes a supervised variation of LLE. Similar to LLE, it retrieves a low-dimensional global coordinate system that faithfully represents the embedded manifold. Unlike LLE, however, it produces an embedding in which predefined modes of variation are preserved. This can improve several supervised learning tasks including pattern recognition, regression, and data visualization.

Stylized synthesis of facial speech motions

AbstractStylized synthesis of facial speech motions is central to facial animation. Most synthesis algorithms put emphasis on the reasonable concatenation of captured motion segments. The dynamic modeling of speech units, e.g. visemes and visyllables (the visual appearance of a syllable), has not drawn much attention. In this paper, we address the fundamental issues regarding the stylized dynamic modeling of visyllables. The decomposable generalized model is learnt for the stylized motion synthesis. The visyllable modeling includes two parts: (1) A dynamic model for each kind of visyllable that is learnt based on a Gaussian Process Dynamical Model; (2) A multilinear model based unified mapping between the high dimensional observation space and low dimensional latent space. The dynamic visyllable model embeds the high dimensional motion data, and constructs the dynamic mapping in the latent space simultaneously. To generalize the visyllable model from several instances, the mapping coefficient matrices are assembled to a tensor, which is decomposed into independent modes, e.g. identity and uttering styles. Therefore, with the linear combination of components in each mode, the novel stylized motions can be synthesized. Copyright © 2007 John Wiley & Sons, Ltd.

Class cover catch digraphs for latent class discovery in gene expression monitoring by DNA microarrays

The purpose of this article is to introduce a data visualization technique for class cover catch digraphs which allows for the discovery of latent subclasses. We illustrate the technique via a pedagogical example and an application to data sets from artificial nose chemical sensing and gene expression monitoring by DNA microarrays. Of particular interest is the discovery of latent subclasses representing chemical concentration in the artificial nose data and two subtypes of acute lymphoblastic leukemia in the gene expression data and the associated conjectures pertaining to the geometry of these subclasses in their respective high-dimensional observation spaces.

強化学習における適応的状態空間構成法

For the application of reinforcement learning to real-world problems, an internal state space has to be constructed from a high dimensional observation space. The algorithm presented here constructs the internal state space during the course of learning desirable actions, and assigns local basis functions adaptively depending on the task requirement. The internal state space initially has only one basis function over the entire observation space, and that basis is eventually divided into smaller ones due to the statistical property of locally weighted temporal difference error. The algorithm was applied to an autonomous robot collision avoidance problem, and the validity of the algorithm was evaluated to show, for instance, the need of a smaller number of basis functions in comparison to other method.

Radial basis functions for exploratory data analysis: An iterative majorisation approach for Minkowski distances based on multidimensional scaling

This paper considers the use of radial basis functions for exploratory data analysis. These are used to model a transformation from a high-dimensional observation space to a low-dimensional one. The parameters of the model are determined by optimising a loss function defined to be the stress function in multidimensional scaling. The metric for the low-dimensional space is taken to be the Minkowski metric with order parameter 1<-p<-2. A scheme based on iterative majorisation is proposed.