Local Learning Method Research Articles

A self-adaptive local metric learning method for classification

Measuring distance among data point pairs is a necessary step among numerous counts of algorithms in machine learning, pattern recognition and data mining. In the local perspective, the emphasis of all existing supervised metric learning algorithms is to shrink similar data points and to separate dissimilar ones in the local neighborhoods. This provides learning more appropriate distance metric in dealing with the within-class multi modal data. In this article, a new supervised local metric learning method named Self-Adaptive Local Metric Learning Method (SA-LM2) has been proposed. The contribution of this method is in five aspects. First, in this method, learning an appropriate metric and defining the radius of local neighborhood are integrated in a joint formulation. Second, unlike the traditional approaches, SA-LM2 learns the parameter of local neighborhood automatically thorough its formulation. As a result, it is a parameter free method, where it does not require any parameters that would need to be tuned. Third, SA-LM2 is formulated as a SemiDefinite Program (SDP) with a global convergence guarantee. Fourth, this method does not need the similar set S, the focus here is on local areas’ data points and their separation from dissimilar ones. Finally, results of SA-LM2 are less influenced by noisy input data points than the other compared global and local algorithms. Results obtained from different experiments indicate the outperformance of this algorithm over its counterparts.

Distance-Dependent Metric Learning

In most existing metric learning methods, the data pairs are equally treated without considering their diversity. In fact, for pairs with different distances, the main purpose of the metric acted on them has some differences. If the pairs have smaller distance, metric should focus more on expanding the negative pairs, which are more easily misjudged. While if the pairs have larger distance, metric should focus more on shrinking the positive pairs. However, most metric learning methods neglect these differences. In this letter, we propose a distance-dependent metric learning (D $^2$ ML) method. It partitions data pairs into different clusters according to the $\ell _2$ distance between them. Each cluster is associated with a Mahalanobis metric that learns the pairs’ distance. This not only allows us to make each metric more targeted and adapt to the data diversity flexibly, but also avoids the problem of computing the distance between points assigned to different clusters, which happens in some local metric learning methods. D $^2$ ML is further extended to D $^3$ ML to embrace the nonlinear capacity of neural network. Experiments on UCI datasets and speaker recognition i-vector machine learning challenge show that the proposed methods are superior to other metric learning methods.

Unsupervised Robust Multiple Kernel Learning via Extracting Local and Global Noises

Kernel-based clustering methods can capture the non-linear structure and identify arbitrarily shaped clusters, so they have been widely used in machine learning tasks. Since the performance of kernel methods critically depends on the choices of kernels, multiple kernel learning methods are proposed to alleviate the effort for kernel designing. The conventional multiple kernel learning methods learn a consensus kernel by linearly combining all candidate kernels, whereas ignoring the influence of the noises. To improve the robustness of multiple kernel learning methods, in this paper, we analyze the local and global noises and design regularized terms to characterize them respectively. Then, we propose a novel local and global de-noising multiple kernel learning method which can explicitly extract the local and global noises to recover the clean kernels for multiple kernel learning. After that, a block coordinate descent algorithm is presented to solve the optimization problem. Finally, the extensive experiments on benchmark data sets will demonstrate the effectiveness of the proposed algorithm.

Semi-supervised dimensionality reduction based on local estimation error

The construction of a graph is extremely important in graph-based semi-supervised learning. However, it is unstable by virtue of sensitivity to the selection of neighbourhood parameter and inaccuracy of the edge weights. Inspired by the good performance of the local learning method, this paper proposes a semi-supervised dimensionality reduction based on local estimation error (LEESSDR) algorithm by utilising local learning projections (LLP) to semi-supervised dimensionality reduction. The algorithm sets the edge weights through minimising the local estimation error and can effectively preserve the global geometric structure as well as the local one of the data. Since LLP does not require its input space to be locally linear, even if it is nonlinear, LLP maps it to the feature space by using kernel functions and then obtains its local estimation error in the feature space. The effectiveness of the proposed method is verified on two popular face databases with promising classification accuracy and favourable robustness.

Target recognition in SAR image based on robust locality discriminant projection

Extracting valuable and discriminative features is one of the crucial issues for target recognition in synthetic aperture radar (SAR) images. In this study, a feature extraction method based on robust locality discriminant projection (RLDP) is presented for SAR target recognition. To characterise the local structural information of SAR images, the manifold learning technique called the supervised locality preserving projection is introduced to learn a linear projection, with which the SAR image can be cast into an implicit feature space. Then, the authors extend t-distributed stochastic neighbour embedding to a parametric framework for optimising the linear projection. In the resulting feature space, the intrinsic neighbour relation with a certain class can be preserved. In addition, the separation between different classes can be enhanced. Unlike most local manifold learning methods, the proposed method is robust to changes of the neighbour parameter. To further analyse the non-linear structure, a useful variant of RLDP named kernel RLDP (KRLDP) is proposed. KRLDP exploits RLDP in an implicit reproducing kernel Hilbert space, where the kernel-based non-linear projection is learned to capture the non-linear structural information. Extensive experiments on moving and stationary target automatic recognition databases demonstrate the effectiveness of the proposed methods.

Novel lightweight odometric learning method for humanoid robot localization

Odometry as one of the inevitable parts of robot behavior control plays an essential role in the localization of humanoid robots. Calculating odometry on the humanoid robots is usually based on one or combination of the vision, laser scanner, magnetic sensor, and pressure sensors. Vision or laser scanner-based approaches require high computational power for analyzing visual information. Hence, these kinds of approaches are not suitable for all kind of the small humanoid robots. On the other hand, it is known that magnetic sensors have instability problems in different environments. Furthermore, calculating accurate dead reckoning (pure odometry) is very difficult because of the complex mechanical system of humanoid robots, the presence of many sources of uncertainty, and inaccuracy in motion executions such as foot slippage. Therefore, this paper presents a robust learning method to localize a humanoid robot named Lightweight Humanoid robot Odometric Learning method (LHOL). This method does not employ any vision, magnetic, additional pressure sensors, and laser scanners, and, therefore, eliminates dependencies to these sensors for the first time. The method's core learning is based on the artificial neural network (ANN) which uses kinematic computations, IMU (roll and pitch data), and the robot's actuators’ internal present load data as input data. The proposed LHOL method proves high accuracy on a novel fully 3D printed kid-sized humanoid robot platform (ARC) with both open-loop and closed-loop walk engines at differently covered floors.

Blind Image Quality Assessment Using Local Consistency Aware Retriever and Uncertainty Aware Evaluator

Blind image quality assessment (BIQA) aims to automatically predict the perceptual quality of a digital image without accessing its pristine reference. Previous studies mainly focus on extracting various quality-relevant image features. By contrast, the explorations on highly efficient learning model are still very limited. Motivated by the fact that it is difficult to approximate a complex and large data set via a global parametric model, we propose a novel local learning method for BIQA to improve quality prediction performance. More specifically, we search for the perceptually similar neighbors of a test image to serve as its unique training set. Unlike the widely used ${k}$ nearest neighbors principle, which only measures the similarity between the testing and training samples, the local consistency of the selected training data is also considered to generate smoother sample space. The image quality is estimated via a sparse Gaussian process. As an additional benefit, the uncertainty of the predicted score is jointly inferred, which can subsequently drive more robust perceptual image processing applications, such as deblocking investigated in this paper. Extensive experiments demonstrate that the proposed learning model leads to consistent quality prediction improvements over many state-of-the-art BIQA algorithms.

Fusion of LLE and stochastic LEM for Persian handwritten digits recognition

In this paper, a new local manifold learning (ML) method is proposed. Our proposed method, which is named FSLL, is based on the fusion of locally linear embedding (LLE) and a new Stochastic Laplacian Eigenmaps (SLEM). SLEM is the same as a common LEM technique, but the coefficients between each data point and its neighbors are calculated by a stochastic process. The coefficients of SLEM make a probability mass function scheme, and their entropy is set to a certain value. The entropy value is an estimation of the locality around each data point. Two criteria will be presented based on the mutual neighborhood conception to determine the entropy value. In LLE, each data point is linearly reconstructed based on its neighbors and then the embedded data manifold is extracted by preserving these linear reconstruction coefficients. LLE and SLEM extract and learn the embedded data manifold by two different kinds of local structure information. In FSLL, two local ML methods, SLEM and LLE, are fused by rewriting their cost functions without the need for any projection space. Fusion of these two techniques provides more structural information at high-dimensional space that can be applied on extracting the embedded low-dimensional data. Also, in this study, a feature vector will be presented by combining a HMAX feature vector and a PCA-based feature vector. Evaluations of the proposed method are done on Persian handwritten digit IFHCDB and IPHD databases in image and feature spaces. The results demonstrate the performance of FSLL and SLEM. The recognition rates are improved about 4% in most dimensionalities. Also, a method of out-of-sample test data extension is proposed corresponding to the proposed methods.

Context-Aware Local Binary Feature Learning for Face Recognition.

In this paper, we propose a context-aware local binary feature learning (CA-LBFL) method for face recognition. Unlike existing learning-based local face descriptors such as discriminant face descriptor (DFD) and compact binary face descriptor (CBFD) which learn each feature code individually, our CA-LBFL exploits the contextual information of adjacent bits by constraining the number of shifts from different binary bits, so that more robust information can be exploited for face representation. Given a face image, we first extract pixel difference vectors (PDV) in local patches, and learn a discriminative mapping in an unsupervised manner to project each pixel difference vector into a context-aware binary vector. Then, we perform clustering on the learned binary codes to construct a codebook, and extract a histogram feature for each face image with the learned codebook as the final representation. In order to exploit local information from different scales, we propose a context-aware local binary multi-scale feature learning (CA-LBMFL) method to jointly learn multiple projection matrices for face representation. To make the proposed methods applicable for heterogeneous face recognition, we present a coupled CA-LBFL (C-CA-LBFL) method and a coupled CA-LBMFL (C-CA-LBMFL) method to reduce the modality gap of corresponding heterogeneous faces in the feature level, respectively. Extensive experimental results on four widely used face datasets clearly show that our methods outperform most state-of-the-art face descriptors.

Unified Simultaneous Clustering and Feature Selection for Unlabeled and Labeled Data.

This paper proposes a novel feature selection method, namely, unified simultaneous clustering feature selection (USCFS). A regularized regression with a new type of target matrix is formulated to select the most discriminative features among the original features from labeled or unlabeled data. The regression with -norm regularization allows the projection matrix to represent an effective selection of discriminative features. For unsupervised feature selection, the target matrix discovers label-like information not from the original data points but rather from projected data points, which are of a reduced dimensionality. Without the aid of an affinity graph-based local structure learning method, USCFS allows the target matrix to capture latent cluster centers via orthogonal basis clustering and to simultaneously select discriminative features guided by latent cluster centers. When class labels are available, the target matrix is also able to find latent class labels by regarding the ground-truth class labels as an approximate guide. Hence, supervised feature selection is realized using these latent class labels, which may differ from the ground-truth class labels. Experimental results demonstrate the effectiveness of the proposed method. Specifically, the proposed method outperforms the state-of-the-art methods on diverse real-world data sets for both the supervised and the unsupervised feature selection.

Microarray Missing Value Imputation: A Regularized Local Learning Method.

Microarray experiments on gene expression inevitably generate missing values, which impedes further downstream biological analysis. Therefore, it is key to estimate the missing values accurately. Most of the existing imputation methods tend to suffer from the over-fitting problem. In this study, we propose two regularized local learning methods for microarray missing value imputation. Motivated by the grouping effect of L2 regularization, after selecting the target gene, we train an L2 Regularized Local Least Squares imputation model (RLLSimpute_L2) on the target gene and its neighbors to estimate the missing values of the target gene. Furthermore, RLLSimpute_L2 imputes the missing values in an ascending order based on the associated missing rate with each target gene. This contributes to fully utilizing the previously estimated values. Besides L2, we further explore L1 regularization and propose an L1 Regularized Local Least Squares imputation model (RLLSimpute_L1). To evaluate their effectiveness, we conducted extensive experimental studies on six benchmark datasets covering both time series and non-time series cases. Nine state-of-the-art imputation methods are compared with RLLSimpute_L2 and RLLSimpute_L1 in terms of three performance metrics. The comparative experimental results indicate that RLLSimpute_L2 outperforms its competitors by achieving smaller imputation errors and better structure preservation of differentially expressed genes.

Discriminative Binary Descriptor for Finger Vein Recognition

In this paper, we propose a new local discriminative feature learning method for finger vein recognition. Unlike most previous finger vein recognition methods, which use hand-crafted descriptors such as local binary pattern, local line binary pattern, and Gabor features, this paper aims to learn a feature mapping to enhance the discriminative ability of local features. To achieve this goal, we first extract multi-directional pixel difference vectors (MDPDVs) for each pixel in a training finger vein image by computing the difference between each pixel and its straight-line neighboring pixels. Second, we learn a feature mapping to project these MDPDVs into low-dimensional binary codes in a supervised manner, where: 1) the loss between the original real-valued codes and learned binary vectors is minimized; 2) the between-class variation of the local binary features is maximized; and 3) the within-class variation of the local binary features is minimized. Last, we represent each finger vein image as a histogram feature by clustering and pooling these binary codes. Experiments on SDUMLA-FV and PolyU databases verify the superior performance of the proposed method over other existing finger vein recognition methods.

Patch-based local learning method for cerebral blood flow quantification with arterial spin-labeling MRI.

Arterial spin-labeling (ASL) perfusion MRI is a non-invasive method for quantifying cerebral blood flow (CBF). Standard ASL CBF calibration mainly relies on pair-wise subtraction of the spin-labeled images and controls images at each voxel separately, ignoring the abundant spatial correlations in ASL data. To address this issue, we previously proposed a multivariate support vector machine (SVM) learning-based algorithm for ASL CBF quantification (SVMASLQ). But the original SVMASLQ was designed to do CBF quantification for all image voxels simultaneously, which is not ideal for considering local signal and noise variations. To fix this problem, we here in this paper extended SVMASLQ into a patch-wise method by using a patch-wise classification kernel. At each voxel, an image patch centered at that voxel was extracted from both the control images and labeled images, which was then input into SVMASLQ to find the corresponding patch of the surrogate perfusion map using a non-linear SVM classifier. Those patches were eventually combined into the final perfusion map. Method evaluations were performed using ASL data from 30 young healthy subjects. The results showed that the patch-wise SVMASLQ increased perfusion map SNR by 6.6% compared to the non-patch-wise SVMASLQ.

An improved Bayesian network method for reconstructing gene regulatory network based on candidate auto selection

BackgroundThe reconstruction of gene regulatory network (GRN) from gene expression data can discover regulatory relationships among genes and gain deep insights into the complicated regulation mechanism of life. However, it is still a great challenge in systems biology and bioinformatics. During the past years, numerous computational approaches have been developed for this goal, and Bayesian network (BN) methods draw most of attention among these methods because of its inherent probability characteristics. However, Bayesian network methods are time consuming and cannot handle large-scale networks due to their high computational complexity, while the mutual information-based methods are highly effective but directionless and have a high false-positive rate.ResultsTo solve these problems, we propose a Candidate Auto Selection algorithm (CAS) based on mutual information and breakpoint detection to restrict the search space in order to accelerate the learning process of Bayesian network. First, the proposed CAS algorithm automatically selects the neighbor candidates of each node before searching the best structure of GRN. Then based on CAS algorithm, we propose a globally optimal greedy search method (CAS + G), which focuses on finding the highest rated network structure, and a local learning method (CAS + L), which focuses on faster learning the structure with little loss of quality.ConclusionResults show that the proposed CAS algorithm can effectively reduce the search space of Bayesian networks through identifying the neighbor candidates of each node. In our experiments, the CAS + G method outperforms the state-of-the-art method on simulation data for inferring GRNs, and the CAS + L method is significantly faster than the state-of-the-art method with little loss of accuracy. Hence, the CAS based methods effectively decrease the computational complexity of Bayesian network and are more suitable for GRN inference.

Identification of predictive cytokine biomarkers of scleroderma via local causal neighborhood methods

Background: Scleroderma is an autoimmune disease with established relationship between immune cytokines and prognosis. Therefore, it is necessary to identify and investigate the causal relationship between cytokines and scleroderma diagnosis and to use this information to identify predictive biomarkers of scleroderma status. Methods: Forty scleroderma positive patients and twenty-four healthy controls have been included in this study. Twenty-nine cytokines implicated in scleroderma have been measured in the bronchoalveolar lavage fluid of these patients, and eight have been found to be univariately associated with scleroderma status. Results: Using local causal neighborhood learning methods, we have found two cytokines, Osteoprotegerin (OPG), also known as osteoclast genesis inhibitory factor, or tumor necrosis factor receptor superfamily member 11B and macrophage inflammatory protein-1 delta, to be causally related to the scleroderma status. Logistic regression predictor based on these cytokines achieves 73% AUC for the task of identifying the scleroderma status. Conclusions: Our results demonstrate the feasibility of developing predictive local causal neighborhood biomarkers of scleroderma status based on bronchoalveolar lavage fluid.

Sparse Bayesian similarity learning based on posterior distribution of data

A major challenge in similarity/distance learning is attaining a strong measure which is close to human notions of similarity. This paper shows why the consideration of data distribution can yield a more effective similarity measure. In addition, the current work both introduces a new scalable similarity measure based on the posterior distribution of data and develops a practical algorithm that learns the proposed measure from the data. To address scalability in this algorithm, the observed data are assumed to have originated from low dimensional latent variables that are close to several subspaces. Other advantages of the currently proposed method include: (1) Providing a principled way to combine metrics in computing the similarity between new instances, unlike local metric learning methods. (2) Automatically identifying the real dimension of latent subspaces, by defining appropriate priors over the parameters of the system via a Bayesian framework. (3) Finding a better projection to low dimensional subspaces, by learning the noise of the latent variables on these subspaces. The present method is evaluated on various real datasets obtained from applications, such as face verification, handwritten digit and spoken letter recognition, network intrusion detection, and image classification. The experimental results confirm that the proposed method significantly outperforms other state-of-the-art metric learning methods on both small and large-scale datasets.

In Silico Prediction of Chemical Toxicity Profile Using Local Lazy Learning

Chemical toxicity is an important reason for late-stage failure in drug R&D. However, it is time-consuming and expensive to identify the multiple toxicities of compounds using the traditional experiments. Thus, it is attractive to build an accurate prediction model for the toxicity profile of compounds. In this study, we carried out a research on six types of toxicities: (I) Acute Toxicity; (II) Mutagenicity; (III) Tumorigenicity; (IV) Skin and Eye Irritation; (V) Reproductive Effects; (VI) Multiple Dose Effects, using local lazy learning (LLL) method for multi-label learning. 17,120 compounds were split into the training set and the test set as a ratio of 4:1 by using the Kennard-Stone algorithm. Four types of properties, including molecular fingerprints (ECFP_4 and FCFP_4), descriptors, and chemical-chemical-interactions, were adopted for model building. The model 'ECFP_4+LLL' yielded the best performance for the test set, while balanced accuracy (BACC) reached 0.692, 0.691, 0.666, 0.680, 0.631, 0.599 for six types of toxicities, respectively. Furthermore, some essential toxicophores for six types of toxicities were identified by using the Laplacian-modified Bayesian model. The accurate prediction model and the chemical toxicophores can provide some guidance for designing drugs with low toxicity.

Simultaneous Local Binary Feature Learning and Encoding for Homogeneous and Heterogeneous Face Recognition.

In this paper, we propose a simultaneous local binary feature learning and encoding (SLBFLE) approach for both homogeneous and heterogeneous face recognition. Unlike existing hand-crafted face descriptors such as local binary pattern (LBP) and Gabor features which usually require strong prior knowledge, our SLBFLE is an unsupervised feature learning approach which automatically learns face representation from raw pixels. Unlike existing binary face descriptors such as the LBP, discriminant face descriptor (DFD), and compact binary face descriptor (CBFD) which use a two-stage feature extraction procedure, our SLBFLE jointly learns binary codes and the codebook for local face patches so that discriminative information from raw pixels from face images of different identities can be obtained by using a one-stage feature learning and encoding procedure. Moreover, we propose a coupled simultaneous local binary feature learning and encoding (C-SLBFLE) method to make the proposed approach suitable for heterogenous face matching. Unlike most existing coupled feature learning methods which learn a pair of transformation matrices for each modality, we exploit both the common and specific information from heterogeneous face samples to characterize their underlying correlations. Experimental results on six widely used face datasets including the LFW, YouTube Face (YTF), FERET, PaSC, CASIA VIS-NIR 2.0, and Multi-PIE datasets are presented to demonstrate the effectiveness of the proposed methods.

Teaching–learning-based optimization with differential and repulsion learning for global optimization and nonlinear modeling

Teaching–learning-based optimization (TLBO) algorithm is one of the recently proposed optimization algorithms. It has been successfully used for solving optimization problems in continuous spaces. To improve the optimization performance of the TLBO algorithm, a modified TLBO algorithm with differential and repulsion learning (DRLTLBO) is presented in this paper. In the proposed algorithm, the differential evolution (DE) operators are introduced into the teacher phase of DRLTLBO to increase the diversity of the new population. In the learner phase of DRLLBO, local learning method or repulsion learning method are adopted according to a certain probability to make learners search knowledge from different directions. In the local learning method, learners learn knowledge not only from the best learner but also from another random learner of their neighbors. In the repulsion learning method, learners learn knowledge from the best learner and keep away from the worst learner of their neighbors. Moreover, self-learning method is adopted to improve the exploitation ability of learners when they are not changed in some continuous generations. To decrease the blindness of random self-learning method, the history information of the corresponding learners in some continuous generations is used in self-learning phase. Furthermore, all learners are regrouped after a certain iterations to improve the local diversity of the learners. In the end, DRLTLBO is tested on 32 benchmark functions with different characteristics and two typical nonlinear modeling problems, and the comparison results show that the proposed DRLTLBO algorithm has shown interesting outcomes in some aspects.

Learning local metrics from pairwise similarity data

We study the problem of learning a similarity function from a set of binary labeled data pairs. A common approach is to learn a similarly function which is a bilinear form associated to the pair of data points. We argue that this class may be too restrictive when handling heterogeneous datasets. To overcome this limitation local metric learning techniques have been advocated in the literature. However, they are subject to certain constraints preventing their usage in many applications. For example, they require knowledge of the class label of the training points. In this paper, we present a local metric learning method, which overcomes these limitations. The method first initializes a Gaussian mixture model on the training data. Then it estimates a set of local metrics and simultaneously refines the mixture’s parameters. Finally, a similarity function is obtained by aggregating the local metrics. We also introduce a novel regularization term, which works well in a transfer learning setting. Our experiments show that the proposed method achieves state-of-the-art results on several real datasets.