Unsupervised Learning Problem Research Articles

Data Exploration with Selection of Representative Regions: Formulation, Axioms, Methods, and Consistency

We present a new type of unsupervised learning problem in which we find a small set of representative regions that approximates a larger dataset. These regions may be presented to a practitioner along with additional information in order to help the practitioner explore the data set. An advantage of this approach is that it does not rely on cluster structure of the data. We formally define this problem, and we present axioms that should be satisfied by functions that measure the quality of representatives. We provide a quality function that satisfies all of these axioms. Using this quality function, we formulate two optimization problems for finding representatives. We provide convergence results for a general class of methods, and we show that these results apply to several specific methods, including methods derived from the solution of the optimization problems formulated in this paper. We provide an example of how representative regions may be used to explore a data set.

Multi-Source Multi-Target Dictionary Learning for Prediction of Cognitive Decline.

Alzheimer's Disease (AD) is the most common type of dementia. Identifying correct biomarkers may determine pre-symptomatic AD subjects and enable early intervention. Recently, Multi-task sparse feature learning has been successfully applied to many computer vision and biomedical informatics researches. It aims to improve the generalization performance by exploiting the shared features among different tasks. However, most of the existing algorithms are formulated as a supervised learning scheme. Its drawback is with either insufficient feature numbers or missing label information. To address these challenges, we formulate an unsupervised framework for multi-task sparse feature learning based on a novel dictionary learning algorithm. To solve the unsupervised learning problem, we propose a two-stage Multi-Source Multi-Target Dictionary Learning (MMDL) algorithm. In stage 1, we propose a multi-source dictionary learning method to utilize the common and individual sparse features in different time slots. In stage 2, supported by a rigorous theoretical analysis, we develop a multi-task learning method to solve the missing label problem. Empirical studies on an N = 3970 longitudinal brain image data set, which involves 2 sources and 5 targets, demonstrate the improved prediction accuracy and speed efficiency of MMDL in comparison with other state-of-the-art algorithms.

Distribution-Preserving Stratified Sampling for Learning Problems.

The need for extracting a small sample from a large amount of real data, possibly streaming, arises routinely in learning problems, e.g., for storage, to cope with computational limitations, obtain good training/test/validation sets, and select minibatches for stochastic gradient neural network training. Unless we have reasons to select the samples in an active way dictated by the specific task and/or model at hand, it is important that the distribution of the selected points is as similar as possible to the original data. This is obvious for unsupervised learning problems, where the goal is to gain insights on the distribution of the data, but it is also relevant for supervised problems, where the theory explains how the training set distribution influences the generalization error. In this paper, we analyze the technique of stratified sampling from the point of view of distances between probabilities. This allows us to introduce an algorithm, based on recursive binary partition of the input space, aimed at obtaining samples that are distributed as much as possible as the original data. A theoretical analysis is proposed, proving the (greedy) optimality of the procedure together with explicit error bounds. An adaptive version of the algorithm is also introduced to cope with streaming data. Simulation tests on various data sets and different learning tasks are also provided.

Learning Sparse Feature Representations Using Probabilistic Quadtrees and Deep Belief Nets

Learning sparse feature representations is a useful instrument for solving an unsupervised learning problem. In this paper, we present three labeled handwritten digit datasets, collectively called n-MNIST by adding noise to the MNIST dataset, and three labeled datasets formed by adding noise to the offline Bangla numeral database. Then we propose a novel framework for the classification of handwritten digits that learns sparse representations using probabilistic quadtrees and Deep Belief Nets. On the MNIST, n-MNIST and noisy Bangla datasets, our framework shows promising results and outperforms traditional Deep Belief Networks.

Chi-square based hierarchical agglomerative clustering for web sessionization

Clustering is one of the fundamental techniques to organise similar objects into proper groups based on features in the domain of data mining, machine learning and pattern recognition. In each cluster, objects are more similar to each other on the basis of particular features. Clustering has numerous applications in multiple domains such as information retrieval, data mining, machine learning, pattern recognition, mathematics, medical and bioinformatics. Web centric applications are expanding day by day and the web has become one of the largest data repositories. During the last decade, information and knowledge retrieval from the web has become a challenging research area. Similarity computation among the data objects (web sessions) is complex, however is a significant problem in unsupervised learning. This research is an attempt to overcome these challenges and problems. The objective of this research paper is to introduce a chi-square based similarity measure to compute the similarity among the sessions. A chi-square based approach is being applied to compute the statistically significant relationship between observed and expected frequencies of the number of pages visited and the time consumed by a user during a session. Moreover, a chi-square based hierarchical agglomerative clustering (Chi-HAC) technique is proposed to extract useful knowledge from web log. The Chi-HAC helps to improve the visualisation of web logs and is equally important for website designers, developers and owners for the improvements of websites at each level. Experimental results with two different log files reveal that the proposed similarity measure with Chi-HAC algorithm has significantly improved the computation among data objects in web sessions.

Automatic Subspace Learning via Principal Coefficients Embedding.

In this paper, we address two challenging problems in unsupervised subspace learning: 1) how to automatically identify the feature dimension of the learned subspace (i.e., automatic subspace learning) and 2) how to learn the underlying subspace in the presence of Gaussian noise (i.e., robust subspace learning). We show that these two problems can be simultaneously solved by proposing a new method [(called principal coefficients embedding (PCE)]. For a given data set , PCE recovers a clean data set from and simultaneously learns a global reconstruction relation of . By preserving into an -dimensional space, the proposed method obtains a projection matrix that can capture the latent manifold structure of , where is automatically determined by the rank of with theoretical guarantees. PCE has three advantages: 1) it can automatically determine the feature dimension even though data are sampled from a union of multiple linear subspaces in presence of the Gaussian noise; 2) although the objective function of PCE only considers the Gaussian noise, experimental results show that it is robust to the non-Gaussian noise (e.g., random pixel corruption) and real disguises; and 3) our method has a closed-form solution and can be calculated very fast. Extensive experimental results show the superiority of PCE on a range of databases with respect to the classification accuracy, robustness, and efficiency.

Unsupervised learning of indoor localization based on received signal strength

AbstractMost indoor wireless sensor network localization methods require costly site surveys to collect fingerprint information for later comparison. Moreover, due to the dynamic nature of fingerprint information in indoor wireless environments, the need for site surveys may be ongoing. In this work, indoor localization is addressed with an unsupervised learning algorithm. Our novel algorithm based on received signal strength combines the information conveyed by both range‐based and range‐free localization with state‐of‐art optimization techniques. A specially designed hierarchical Bayesian hidden Markov model coupled with a particle filter helps mitigate non‐line‐of‐sight and multipath errors. This grid‐based data sample process, derived from the theory of Dirichlet processes, simplifies the global optimization problem of unsupervised learning by employing a single initial hyper‐parameter. Meanwhile, for obtaining accurate coordinates of mobile nodes, a unique semidefinite programming method is used to provide feedback to the radio propagation model. This feedback step can enable the grid‐based algorithms not only to establish the coordinates of a mobile node, but also to optimize the accuracy iteratively. Theoretical and experimental analyses indicate that the proposed algorithm can achieve better localization accuracy than conventional range‐based algorithms without adding computation cost. Copyright © 2016 John Wiley & Sons, Ltd.

Unsupervised dictionary learning with Fisher discriminant for clustering

In this paper, we propose a novel Fisher discriminant unsupervised dictionary learning (FD-UDL) approach, for improving the clustering performance of state-of-the-art dictionary learning approaches in unsupervised scenarios. This is achieved by employing a novel Fisher discriminant criterion on dictionary elements to encourage the diversity between different sub-dictionaries, and also the coherence within each sub-dictionary. Such a discriminant is incorporated to formulate the optimization problem of unsupervised dictionary learning. Furthermore, we provide an analytical solution to the proposed optimization problem, obtaining the learned dictionary for clustering tasks. Unlike previous approaches for unsupervised clustering, the proposed FD-UDL approach takes into account both within-class and between-class scatters of sub-dictionaries, rather than only considering diversity between different sub-dictionaries. Finally, experiments on synthetic data, face and handwritten digit clustering tasks show the improved clustering accuracy over other state-of-the-art dictionary learning and clustering approaches.

Adaptive Multi-Subpopulation Competition and Multi-Niche Crowding based Memetic Algorithm for Automatic Data Clustering

Automatic data clustering, whose goal is to recover the proper number of clusters as well as appropriate partitioning of data sets, is a fundamental yet challenging problem in unsupervised learning. In this paper, adaptive multisubpopulation competition (AMC) and multiniche crowding are proposed and incorporated into a memetic algorithm to tackle the problem. The AMC mechanism is developed to ensure a diverse search over solution subspaces corresponding to different numbers of clusters while allowing more promising subspaces to be more intensively searched. In this mechanism, the amount of individuals to be migrated between subpopulations is adaptively controlled according to the performance of subpopulations as well as the diversity of cluster numbers in population. Further, the migration is restricted to occur between subpopulations with relatively similar performances. Additionally, subpopulations with different performances are devised to search their corresponding subspaces with different exploration powers. The adaptive multiniche crowding scheme is designed to promote a diverse search of the subspace while allowing an efficient convergence of the corresponding subpopulation. This is achieved by dynamically adjusting parameter values of a multiniche crowding method to form and maintain diverged niches of high fitness within the subpopulation. The performance of proposed algorithm has been demonstrated through a series of experiments on both artificial and real data, and compared with existing methods. The results reveal that our proposed algorithm can achieve superior clustering performance and outperform related methods.

Discriminative Transfer Subspace Learning via Low-Rank and Sparse Representation.

In this paper, we address the problem of unsupervised domain transfer learning in which no labels are available in the target domain. We use a transformation matrix to transfer both the source and target data to a common subspace, where each target sample can be represented by a combination of source samples such that the samples from different domains can be well interlaced. In this way, the discrepancy of the source and target domains is reduced. By imposing joint low-rank and sparse constraints on the reconstruction coefficient matrix, the global and local structures of data can be preserved. To enlarge the margins between different classes as much as possible and provide more freedom to diminish the discrepancy, a flexible linear classifier (projection) is obtained by learning a non-negative label relaxation matrix that allows the strict binary label matrix to relax into a slack variable matrix. Our method can avoid a potentially negative transfer by using a sparse matrix to model the noise and, thus, is more robust to different types of noise. We formulate our problem as a constrained low-rankness and sparsity minimization problem and solve it by the inexact augmented Lagrange multiplier method. Extensive experiments on various visual domain adaptation tasks show the superiority of the proposed method over the state-of-the art methods. The MATLAB code of our method will be publicly available at http://www.yongxu.org/lunwen.html.

An in silico model to demonstrate the effects of Maspin on cancer cell dynamics

Most cancer treatments efficacy depends on tumor metastasis suppression, where tumor suppressor genes play an important role. Maspin (Mammary Serine Protease Inhibitor), an non-inhibitory serpin has been reported as a potential tumor suppressor to influence cell migration, adhesion, proliferation and apoptosis in in vitro and in vivo experiments in last two decades. Lack of computational investigations hinders its ability to go through clinical trials. Previously, we reported first computational model for maspin effects on tumor growth using artificial neural network and cellular automata paradigm with in vitro data support. This paper extends the previous in silico model by encompassing how maspin influences cell migration and the cell–extracellular matrix interaction in subcellular level. A feedforward neural network was used to define each cell behavior (proliferation, quiescence, apoptosis) which followed a cell-cycle algorithm to show the microenvironment impacts over tumor growth. Furthermore, the model concentrates how the in silico experiments results can further confirm the fact that maspin reduces cell migration using specific in vitro data verification method. The data collected from in vitro and in silico experiments formulates an unsupervised learning problem which can be solved by using different clustering algorithms. A density based clustering technique was developed to measure the similarity between two datasets based on the number of links between instances. Our proposed clustering algorithm first finds the nearest neighbors of each instance, and then redefines the similarity between pairs of instances in terms of how many nearest neighbors share the two instances. The number of links between two instances is defined as the number of common neighbors they have. The results showed significant resemblances with in vitro experimental data. The results also offer a new insight into the dynamics of maspin and establish as a metastasis suppressor gene for further molecular research.

Domain Adaptation Extreme Learning Machines for Drift Compensation in E-Nose Systems

This paper addresses an important issue, known as sensor drift that behaves a nonlinear dynamic property in electronic nose (E-nose), from the viewpoint of machine learning. Traditional methods for drift compensation are laborious and costly due to the frequent acquisition and labeling process for gases samples recalibration. Extreme learning machines (ELMs) have been confirmed to be efficient and effective learning techniques for pattern recognition and regression. However, ELMs primarily focus on the supervised, semi-supervised and unsupervised learning problems in single domain (i.e. source domain). To our best knowledge, ELM with cross-domain learning capability has never been studied. This paper proposes a unified framework, referred to as Domain Adaptation Extreme Learning Machine (DAELM), which learns a robust classifier by leveraging a limited number of labeled data from target domain for drift compensation as well as gases recognition in E-nose systems, without loss of the computational efficiency and learning ability of traditional ELM. In the unified framework, two algorithms called DAELM-S and DAELM-T are proposed for the purpose of this paper, respectively. In order to percept the differences among ELM, DAELM-S and DAELM-T, two remarks are provided. Experiments on the popular sensor drift data with multiple batches collected by E-nose system clearly demonstrate that the proposed DAELM significantly outperforms existing drift compensation methods without cumbersome measures, and also bring new perspectives for ELM.

Unsupervised Object Class Discovery via Saliency-Guided Multiple Class Learning.

In this paper, we tackle the problem of common object (multiple classes) discovery from a set of input images, where we assume the presence of one object class in each image. This problem is, loosely speaking, unsupervised since we do not know a priori about the object type, location, and scale in each image. We observe that the general task of object class discovery in a fully unsupervised manner is intrinsically ambiguous; here we adopt saliency detection to propose candidate image windows/patches to turn an unsupervised learning problem into a weakly-supervised learning problem. In the paper, we propose an algorithm for simultaneously localizing objects and discovering object classes via bottom-up (saliency-guided) multiple class learning (bMCL). Our contributions are three-fold: (1) we adopt saliency detection to convert unsupervised learning into multiple instance learning, formulated as bottom-up multiple class learning (bMCL); (2) we propose an integrated framework that simultaneously performs object localization, object class discovery, and object detector training; (3) we demonstrate that our framework yields significant improvements over existing methods for multi-class object discovery and possess evident advantages over competing methods in computer vision. In addition, although saliency detection has recently attracted much attention, its practical usage for high-level vision tasks has yet to be justified. Our method validates the usefulness of saliency detection to output "noisy input" for a top-down method to extract common patterns.

Semi-Supervised Local Fisher Discriminant Analysis Based on Reconstruction Probability Class

Fisher discriminant analysis (FDA) is a classic supervised dimensionality reduction method in statistical pattern recognition. FDA can maximize the scatter between different classes, while minimizing the scatter within each class. As it only utilizes the labeled data and ignores the unlabeled data in the analysis process of FDA, it cannot be used to solve the unsupervised learning problems. Its performance is also very poor in dealing with semi-supervised learning problems in some cases. Recently, several semi-supervised learning methods as an extension of FDA have proposed. Most of these methods solve the semi-supervised problem by using a tradeoff parameter that evaluates the ratio of the supervised and unsupervised methods. In this paper, we propose a general semi-supervised dimensionality learning idea for the partially labeled data, namely the reconstruction probability class of labeled and unlabeled data. Based on the probability class optimizes Fisher criterion function, we propose a novel Semi-Supervised Local Fisher Discriminant Analysis (S2LFDA) method. Experimental results on real-world datasets demonstrate its effectiveness compared to the existing similar correlation methods.

Learning Poisson Binomial Distributions

We consider a basic problem in unsupervised learning: learning an unknown Poisson binomial distribution. A Poisson binomial distribution (PBD) over $$\{0,1,\ldots ,n\}$${0,1,?,n} is the distribution of a sum of $$n$$n independent Bernoulli random variables which may have arbitrary, potentially non-equal, expectations. These distributions were first studied by Poisson (Recherches sur la Probabilite des jugements en matie criminelle et en matiere civile. Bachelier, Paris, 1837) and are a natural $$n$$n-parameter generalization of the familiar Binomial Distribution. Surprisingly, prior to our work this basic learning problem was poorly understood, and known results for it were far from optimal. We essentially settle the complexity of the learning problem for this basic class of distributions. As our first main result we give a highly efficient algorithm which learns to $$\epsilon $$∈-accuracy (with respect to the total variation distance) using $$\tilde{O}(1/ \epsilon ^{3})$$O~(1/∈3) samples independent of$$n$$n. The running time of the algorithm is quasilinear in the size of its input data, i.e., $$\tilde{O}(\log (n)/\epsilon ^{3})$$O~(log(n)/∈3) bit-operations (we write $$\tilde{O}(\cdot )$$O~(·) to hide factors which are polylogarithmic in the argument to $$\tilde{O}(\cdot )$$O~(·); thus, for example, $$\tilde{O}(a \log b)$$O~(alogb) denotes a quantity which is $$O(a \log b \cdot \log ^c(a \log b))$$O(alogb·logc(alogb)) for some absolute constant $$c$$c. Observe that each draw from the distribution is a $$\log (n)$$log(n)-bit string). Our second main result is a proper learning algorithm that learns to $$\epsilon $$∈-accuracy using $$\tilde{O}(1/\epsilon ^{2})$$O~(1/∈2) samples, and runs in time $$(1/\epsilon )^{\mathrm {poly}(\log (1/\epsilon ))} \cdot \log n$$(1/∈)poly(log(1/∈))·logn. This sample complexity is nearly optimal, since any algorithm for this problem must use $$\Omega (1/\epsilon ^{2})$$Ω(1/∈2) samples. We also give positive and negative results for some extensions of this learning problem to weighted sums of independent Bernoulli random variables.

An unsupervised learning algorithm for membrane computing

This paper focuses on the unsupervised learning problem within membrane computing, and proposes an innovative solution inspired by membrane computing techniques, the fuzzy membrane clustering algorithm. An evolution–communication P system with nested membrane structure is the core component of the algorithm. The feasible cluster centers are represented by means of objects, and three types of membranes are considered: evolution, local store, and global store. Based on the designed membrane structure and the inherent communication mechanism, a modified differential evolution mechanism is developed to evolve the objects in the system. Under the control of the evolution–communication mechanism of the P system, the proposed fuzzy clustering algorithm achieves good fuzzy partitioning for a data set. The proposed fuzzy clustering algorithm is compared to three recently-developed and two classical clustering algorithms for five artificial and five real-life data sets.

Decision Theoretic Evaluation of Rough Fuzzy Clustering

Clustering is the process of organizing dissimilar objects into natural groups in such a way objects in the same group is more similar than objects in the different groups. Since we know clustering is an unsupervised learning problem, typical clustering algorithms not achieving its end to handle uncertainty that exists in the real life experience. Though fuzzy clustering handles incompleteness and vagueness in the data set efficiently, it is highly descriptive than hard clustering algorithm. Rough clustering algorithm is the popular soft clustering technique which uses rough set to handle uncertainty. In Rough Fuzzy clustering, each cluster is represented by centroid, crisp lower approximation and fuzzy boundary. Clustering undergoes sequence of partitions where cluster evaluation is the final step in clustering process. Efficient clustering structure can be obtained through validity measures. Various validity measures have been proposed to evaluate rough fuzzy clustering. Since those measures are Geometric measures, this paper proposes decision theoretic measure for validating rough fuzzy clustering structure.

Cell population identification using fluorescence-minus-one controls with a one-class classifying algorithm.

The tried and true approach of flow cytometry data analysis is to manually gate on each biomarker separately, which is feasible for a small number of biomarkers, e.g. less than five. However, this rapidly becomes confusing as the number of biomarker increases. Furthermore, multivariate structure is not taken into account. Recently, automated gating algorithms have been implemented, all of which rely on unsupervised learning methodology. However, all unsupervised learning outputs suffer the same difficulties in validation in the absence of external knowledge, regardless of application domain. We present a new semi-automated algorithm for population discovery that is based on comparison to fluorescence-minus-one controls, thus transferring the problem into that of one-class classification, as opposed to being an unsupervised learning problem. The novel one-class classification algorithm is based on common principal components and can accommodate complex mixtures of multivariate densities. Computational time is short, and the simple nature of the calculations means the algorithm can easily be adapted to process large numbers of cells (10(6)). Furthermore, we are able to find rare cell populations as well as populations with low biomarker concentration, both of which are inherently hard to do in an unsupervised learning context without prior knowledge of the samples' composition. R scripts are available via https://fccf.mpiib-berlin.mpg.de/daten/drfz/bioinformatics/with{username,password}={bioinformatics,Sar=Gac4}.

Ensemble learning with trees and rules: Supervised, semi-supervised, unsupervised

In this article, we propose several new approaches for post processing a large ensemble of conjunctive rules for supervised, semi-supervised and unsupervised learning problems. We show with various examples that for high dimensional regression problems the models constructed by post processing the rules with partial least squares regression have significantly better prediction performance than the ones produced by the random forest or the rulefit algorithms which use equal weights or weights estimated from lasso regression. When rule ensembles are used for semi-supervised and unsupervised learning, the internal and external measures of cluster validity point to high quality groupings.

A Spectral Algorithm for Latent Dirichlet Allocation

Topic modeling is a generalization of clustering that posits that observations (words in a document) are generated by multiple latent factors (topics), as opposed to just one. The increased representational power comes at the cost of a more challenging unsupervised learning problem for estimating the topic-word distributions when only words are observed, and the topics are hidden. This work provides a simple and efficient learning procedure that is guaranteed to recover the parameters for a wide class of multi-view models and topic models, including latent Dirichlet allocation (LDA). For LDA, the procedure correctly recovers both the topic-word distributions and the parameters of the Dirichlet prior over the topic mixtures, using only trigram statistics (i.e., third order moments, which may be estimated with documents containing just three words). The method is based on an efficiently computable orthogonal tensor decomposition of low-order moments.