Manifold Regularization Framework Research Articles

Discriminative sparse subspace learning with manifold regularization

Common subspace learning methods only utilize local or global structure in feature extraction, and cannot obtain the global optimal discriminative projection matrix. For this reason, this paper proposes a discriminative sparse subspace learning method based on the manifold regularization framework (DSSL-MR), which introduces the graph Laplacian matrix that reflects the intrinsic geometric structure of the sample as a penalty term. DSSL-MR simultaneously uses both sub-manifold and multi-manifold information of samples for obtaining optimal projection to enhance the discriminability of different classes in subspace. DSSL-MR uses the sparse property of the L2,1-norm to constrain the projection matrix, which can eliminate redundant features and select features that are significant for classification. It is a linear supervised method, which belongs to the Fisher discriminant analysis framework. Experimental results on multiple real-world datasets show that the algorithm is very effective in classification and has high recognition rates.

Transductive transfer broad learning for cross-domain information exploration and multigrade soft sensor application

Without sufficient labeled data, the construction of accurate soft-sensor models for multigrade chemical processes is challenging. To alleviate the dilemma, a transductive transfer broad learning (TTBL) model for cross-domain information exploration is proposed. The features are extracted by the nodes of broad learning system. Then, similar sample information from the current and related domains is captured by the k nearest-neighbor graph and retained by the manifold regularization framework. By exploring the available cross-domain information, unlabeled data in the prediction domain can be utilized for modeling. Finally, a TTBL model is constructed assisted by the fast leave-one-out cross-validation strategy. TTBL can effectively exploit the useful information of cross-domain data to improve prediction performance. Experimental results on two multigrade chemical processes demonstrate its superiorities compared with several traditional methods.

A Framework for Distributed Semi-supervised Learning Using Single-layer Feedforward Networks

This paper aims to propose a framework for manifold regularization (MR) based distributed semi-supervised learning (DSSL) using single layer feed-forward neural network (SLFNN). The proposed framework, denoted as DSSL-SLFNN is based on the SLFNN, MR framework, and distributed optimization strategy. Then, a series of algorithms are derived to solve DSSL problems. In DSSL problems, data consisting of labeled and unlabeled samples are distributed over a communication network, where each node has only access to its own data and can only communicate with its neighbors. In some scenarios, DSSL problems cannot be solved by centralized algorithms. According to the DSSL-SLFNN framework, each node over the communication network exchanges the initial parameters of the SLFNN with the same basis functions for semi-supervised learning (SSL). All nodes calculate the global optimal coefficients of the SLFNN by using distributed datasets and local updates. During the learning process, each node only exchanges local coefficients with its neighbors rather than raw data. It means that DSSL-SLFNN based algorithms work in a fully distributed fashion and are privacy preserving methods. Finally, several simulations are presented to show the efficiency of the proposed framework and the derived algorithms.

Online Semisupervised Broad Learning System for Industrial Fault Diagnosis

Recently, broad learning system (BLS) has been introduced to solve industrial fault diagnosis problems and has achieved impressive performance. As a flat network, BLS enjoys a simple linear structure, which enables BLS to train and update the model efficiently in an incremental manner, and it potentially has better generalization capacity than deep learning methods when training data are limited. The basic BLS is a supervised learning method that requires all the training data to be labeled. However, in many practical industrial scenarios, data labels are usually difficult to obtain. Existing semisupervised variant uses manifold regularization framework to capture the information of unlabeled data, however, such a method will sacrifice the incremental learning capacity of BLS. Considering that in many practical applications, training data are sequentially generated, in this article, an online semisupervised broad learning system (OSSBLS) is proposed for fault diagnosis in these cases. The proposed method not only can efficiently construct and incrementally update the model, but also can take advantage of unlabeled data to improve the model's diagnostic performance. Experimental results on the Tennessee Eastman process and a real-world air compressor working process demonstrate the superiority of OSSBLS in terms of both diagnostic performance and time consumption.

Enhancing performance of the back-propagation algorithm based on a novel regularization method of preserving inter-object-distance of data

Artificial neural networks, consisting of many levels of nonlinearities, have been widely used to deal with various supervised learning tasks. At present, the most popular and effective training method is back-propagation algorithm (BP). Inspired by manifold regularization framework, we introduce a novel regularization framework, which aims at preserving the inter-object-distance of the data. Then a refined BP algorithm (IOD-BP) is proposed by imposing the proposed regularization framework into the objective function of BP algorithm. Comparative experiments on various benchmark classification tasks show that the new regularization BP method significantly improves the performance of BP algorithm in terms of classification accuracy.

Transfer of semi-supervised broad learning system in electroencephalography signal classification

Electroencephalography (EEG) signal classification is a crucial part in motor imagery brain–computer interface (BCI) system. Traditional supervised learning methods have performed well pleasing in EEG classification. Unfortunately, the unlabeled samples are easier to collect than labeled samples. In addition, recent studies have shown that it may degenerate performance of semi-supervised learning by exploiting unlabeled samples without selection. To address these issues, a novel semi-supervised broad learning system with transfer learning (TSS-BLS) is proposed in this paper. First, the pseudo-labels of unlabeled samples are obtained using the joint distribution adaptation algorithm. TSS-BLS is then constructed by an improved manifold regularization framework containing both labeled and pseudo-label information. Finally, the effectiveness of the proposed TSS-BLS is evaluated on three BCI competition datasets and four benchmark datasets from UCI repository and compared with seven state-of-the-art algorithms, including ELM, SS-ELM, HELM, SVM, LapSVM, BLS and GSS-BLS. Experimental results show that the performance of TSS-BLS is superior to BLS and GSS-BLS on average. It is thereby shown that TSS-BLS is safe and efficient for EEG classification.

Hyperspectral image classification with discriminative manifold broad learning system

It has been proved that hyperspectral image (HSI) classification task benefits from introducing additional spatial information. However, how to classify high-dimensional hyperspectral images under the condition of limited training samples is still a challenge. In this paper, we propose a hyperspectral image classification framework based on discriminant manifold broad learning system (DMBLS). We introduce the manifold structure and discriminant information of the data samples as prior knowledge into broad learning system (BLS), which effectively solves the problem of insufficient learning caused by BLS with limited training samples. Firstly we construct two different types of manifold structures for class modelling: the intra-class manifold structures and the inter-class ones. Secondly, we integrate these local discriminant manifold structures to build a manifold regularization framework. Finally, we add this framework to the DMBLS via minimizing the intra-class manifold structures, while maximizing the inter-class ones. And we think this operation can optimize the projection direction of output weights in the DMBLS, enhancing the discrimination ability of these weights. Experimental results on three benchmark HSI datasets show that DMBLS can effectively improve the classification accuracy of hyperspectral images under the condition of limited training samples.

Improving the predictions of soil properties from VNIR–SWIR spectra in an unlabeled region using semi-supervised and active learning

Monitoring the status of the soil ecosystem to identify the spatio-temporal extent of the pressures exerted and mitigate the effects of climate change and land degradation necessitates the need for reliable and cost-effective solutions. To address this need, soil spectroscopy in the visible, near- and shortwave-infrared (VNIR–SWIR) has emerged as a viable alternative to traditional analytical approaches. To this end, large-scale soil spectral libraries coupled with advanced machine learning tools have been developed to infer the soil properties from the hyperspectral signatures. However, models developed from one region may exhibit diminished performance when applied to a new, unseen by the model, region due to the large and inherent soil variability (e.g. pedogenetical differences, diverse soil types etc.). Given an existing spectral library with labeled data and a new unlabeled region (i.e. where no soil samples are analytically measured) the question then becomes how to best develop a model which can more accurately predict the soil properties of the unlabeled region.In this paper, a machine learning technique leveraging on the capabilities of semi-supervised learning which exploits the predictors’ distribution of the unlabeled dataset and of active learning which expertly selects a small set of data from the unlabeled dataset as a spiking subset in order to develop a more robust model is proposed. The semi-supervised learning approach is the Laplacian Support Vector Regression following the manifold regularization framework. As far as the active learning component is concerned, the pool-based approach is utilized as it best matches with the aforementioned use-case scenario, which iteratively selects a subset of data from the unlabeled region to spike the calibration set. As a query strategy, a novel machine learning–based strategy is proposed herein to best identify the spiking subset at each iteration. The experimental analysis was conducted using data from the Land Use and Coverage Area Frame Survey of 2009 which covered most of the then member-states of the European Union, and in particular by focusing on the mineral cropland soil samples from 5 different countries. The statistical analysis conducted ascertained the efficacy of our approach when compared to the current state-of-the-art in soil spectroscopy.

Semi-Supervised Broad Learning System Based on Manifold Regularization and Broad Network

Broad Learning System (BLS) are widely used in many fields because of its strong feature extraction ability and high computational efficiency. However, the BLS is mainly used in supervised learning, which greatly limits the applicability of the BLS. And the obtained data is less labeled data, but is a large number of unlabeled data. Therefore, the BLS is extended based on the semi-supervised learning of manifold regularization framework to propose a semi-supervised broad learning system (SS-BLS). Firstly, the features are extracted from labeled and unlabeled data by building feature nodes and enhancement nodes. Then the manifold regularization framework is used to construct Laplacian matrix. Next, the feature nodes, enhancement nodes and Laplacian matrix are combined to construct the objective function, which is effectively solved by ridge regression in order to obtain the output coefficients. Finally, the validity of the SS-BLS is verified by three different complex data of G50C, MNIST, and NORB, respectively. The experiment result show that the SS-BLS can achieve higher classification accuracy for different complex data, takes on fast operation speed and strong generalization ability.

Manifold regularization based distributed semi-supervised learning algorithm using extreme learning machine over time-varying network

This paper aims to propose a distributed semi-supervised learning (SSL) algorithm using extreme learning machine (ELM) over time-varying communication network, whose topology changes over time rather than being fixed. In distributed SSL problems, training data including labeled and unlabeled samples is separately stored on each node over the communication network and cannot be centrally processed. In order to solve these problems, we propose an algorithm combining the semi-supervised ELM (SS-ELM) algorithm with the zero-gradient-sum (ZGS) distributed optimization strategy. The SS-ELM algorithm, based on the manifold regularization (MR) framework, is used to approximate the mapping of samples on each node over the communication network. Then, the ZGS strategy is used to train the globally optimal coefficients of the single layer feed-forward neural network (SLFNN) corresponding to the SS-ELM algorithm. Thus, we denote the proposed algorithm as the distributed SS-ELM (DSS-ELM) algorithm. During the training process, all nodes over the communication network exchange updated coefficients rather than raw data with their neighboring nodes. It means that the DSS-ELM algorithm is a fully distributed and privacy-preserving algorithm. The convergence of the proposed DSS-ELM is guaranteed by the Lyapunov method. At last, some simulations are presented to show the efficiency of the proposed algorithm.

Causal Learning via Manifold Regularization.

This paper frames causal structure estimation as a machine learning task. The idea is to treat indicators of causal relationships between variables as 'labels' and to exploit available data on the variables of interest to provide features for the labelling task. Background scientific knowledge or any available interventional data provide labels on some causal relationships and the remainder are treated as unlabelled. To illustrate the key ideas, we develop a distance-based approach (based on bivariate histograms) within a manifold regularization framework. We present empirical results on three different biological data sets (including examples where causal effects can be verified by experimental intervention), that together demonstrate the efficacy and general nature of the approach as well as its simplicity from a user's point of view.

Video Salient Object Detection via Robust Seeds Extraction and Multi-graphs Manifold Propagation

Video salient object detection aims at distinguishing the salient objects from the complex background and highlighting them uniformly in the spatiotemporal domain, which still suffers from the interference of the complicated dynamic background in unconstrained videos. To address this problem, we propose a novel coarse-to-fine spatiotemporal salient object detection method. Specifically, we first model a novel motion energy to exclude the motion noise by exploiting the motion magnitude and motion orientation. Then, a supervoxel-level inter-frame graph model is constructed for each pair of adjacent frames independently, and a robust graph clustering-based saliency seed generation method is proposed to produce a coarse saliency map. Furthermore, the supervoxel-level inter-frame graph model is reconstructed by considering the regional spatiotemporal consistency constraint based on the coarse saliency map. The prior information obtained from pixel clustering is also taken into account to optimize the weight of the inter-frame graph model. Finally, a multi-graphs saliency propagation method is exploited under the manifold regularization framework by fusing the motion energy and appearance feature to refine the coarse saliency map. The extensive experiments on two widely used datasets validate the effectiveness and superiority of the proposed method against 13 state-of-the-art methods in terms of PR-curves, scores of S-measure, $F_{\beta }$ , and MAE.

Semi-supervised classification of multiple kernels embedding manifold information

For semi-supervised learning, we propose the Laplacian embedded multiple kernel regression model. As we incorporate the multiple kernel occasion into manifold regularization framework, the models we proposed are flexible in many kinds of datasets and have a solid theoretical foundation. The proposed model can solve the two problems, which are the computation cost of manifold regularization framework and the difficulty in dealing with multi-source or multi-attribute datasets. Though manifold regularization is a convex optimization formulation, it often leads to dense matrix inversion with computation cost. Laplacian embedded method we adopted can solve the problem, however it lacks the proper ability to process complex datasets. Therefore, we further use multiple kernel learning as a part of the proposed model to strengthen its ability. Experiments on several datasets compared with the state-of-the-art methods show the effectiveness and efficiency of the proposed model.

Updating initial labels from spectral graph by manifold regularization for saliency detection

This paper presents a novel saliency detection method via updating initial labels from spectral graph in a semi-supervised learning (SSL) framework. For updating labels efficiently with graph-based SSL, two principles generally should be considered. The first one is that the updated labels should not change too much from their initial assignment. The second one is that the updated labels should not change too much between similar samples. To follow the first principle, the biggest eigenvector of Laplacian matrix, which contains rich contrast between background regions and salient regions, is employed to obtain the initial label vector. To follow the second principle, a new graph construction scheme, in which only boundary samples with similar features can be connected with each other, is proposed to reduce the geodesic distance in graph. Then a graph-based manifold regularization framework is exploited to update the label vector for separating salient samples from non-salient samples. A refinement function cooperating with an activation function is further presented for saliency optimization. Experimental results show that the proposed method achieves competitive performance against some recent state-of-the-art algorithms for saliency detection.

Robust Semi-Supervised Classification for Noisy Labels Based on Self-Paced Learning

Data labeling is a tedious and subjective task that can be time consuming and error-prone; however, most learning algorithms are sensitive to noisy labels. This problem raises the need to develop algorithms that can exploit large amount of unlabeled data and also be robust to noisy label information. In this letter, we propose a novel semi-supervised classification framework that is robust to noisy labels, named self-paced manifold regularization. The proposed framework naturally integrates self-paced learning regime into the manifold regularization framework for selecting labeled training samples in a theoretically sound manner, and utilizes locally linear reconstructions to control the smoothness of the classifier with respect to the manifold structure of data. Finally, the alternative search strategy is adopted for the proposed framework to obtain the classifier. The proposed method can not only suppress the negative effect of noisy initial labels in semi-supervised learning, but also obtain an explicit multiclass classifier for newly coming data points. Experimental results demonstrate the effectiveness of the proposed method.

Enhanced manifold regularization for semi-supervised classification.

Manifold regularization (MR) has become one of the most widely used approaches in the semi-supervised learning field. It has shown superiority by exploiting the local manifold structure of both labeled and unlabeled data. The manifold structure is modeled by constructing a Laplacian graph and then incorporated in learning through a smoothness regularization term. Hence the labels of labeled and unlabeled data vary smoothly along the geodesics on the manifold. However, MR has ignored the discriminative ability of the labeled and unlabeled data. To address the problem, we propose an enhanced MR framework for semi-supervised classification in which the local discriminative information of the labeled and unlabeled data is explicitly exploited. To make full use of labeled data, we firstly employ a semi-supervised clustering method to discover the underlying data space structure of the whole dataset. Then we construct a local discrimination graph to model the discriminative information of labeled and unlabeled data according to the discovered intrinsic structure. Therefore, the data points that may be from different clusters, though similar on the manifold, are enforced far away from each other. Finally, the discrimination graph is incorporated into the MR framework. In particular, we utilize semi-supervised fuzzy c-means and Laplacian regularized Kernel minimum squared error for semi-supervised clustering and classification, respectively. Experimental results on several benchmark datasets and face recognition demonstrate the effectiveness of our proposed method.

基于仿射子空间稀疏表示的半监督分类

Graph-based semi-supervised classification is one of the hottest research areas in machine learning and data mining. These methods usually model an entire dataset as a graph, then utilize the structure information extracted by the graph to help with the classification of unlabeled data. Generally speaking, the performance of graph-based semi-supervised classification methods highly depends on the constructed graphs. In this paper, we propose a new kind of graph construction method based on affine subspace sparse representation. The proposed sparse coding method minimizes the construction error of the input signal, considering three constraints: (1) the input signal being approximately reconstructed by the affine combination of the dictionary; (2) the nonnegativity constraint of the reconstruction coefficients; (3) the sparsity constraint of the reconstruction coefficients. Based on the constraints, we present the ι 0-norm constrained optimization problem for sparse coding; then, we propose the algorithm to solve the problem and further construct the ι 0-graph of data. Finally, under the manifold regularization framework, we propose a new kind of semi-supervised classification method by introducing the regularization term that measures the structure preserving error of the ι 0-graph. The proposed semi-supervised classification method has an explicit multiclass classification function and inherits the strong discriminative information from sparse representation. As a result, it has efficient and effective classification ability. Experimental results on artificial and real-world datasets are provided to show the effectiveness of the proposed method.

Denoising Laplacian multi-layer extreme learning machine

Most of semi-supervised learning algorithms based on manifold regularization framework are surface learning algorithms, such as semi-supervised ELM (SS-ELM) and Laplacian smooth twin support vector machine (Lap-STSVM). Multi-layer extreme learning machine (ML-ELM) stacks extreme learning machine based auto encoder (ELM-AE) to create a multi-layer neural network. ML-ELM not only approximates the complicated function but also achieves fast training time. The outputs of ELM-AE are the same as inputs, which cannot guarantee the effectiveness of the learning feature representations. We put forward extreme learning machine based denoising auto encoder (ELM-DAE) which introduces local denoising criterion into ELM-AE and is used as the basic component for Denoising ML-ELM. Resembling ML-ELM, Denoising ML-ELM stacks ELM-DAE to create a deep network. And then we introduce manifold regularization into the model of Denoising ML-ELM and propose denoising Laplacian ML-ELM (Denoising Lap-ML-ELM). Denoising Lap-ML-ELM is more efficient than SS-ELM in classification and does not need to spend too much time. Experimental results show that Denoising ML-ELM and Denoising Lap-ML-ELM are effective learning algorithms.

An Efficient Semi-Supervised Classifier Based on Block-Polynomial Mapping

In this paper, we propose a block-polynomial mapping for image feature learning, which can be efficiently represented by the matrix Khatri-Rao product. The block-polynomial mapping not only captures the local discriminative information within the image structure, but is also much more efficient than the traditional kernel mapping. Moreover, we embed the proposed mapping into the manifold regularization framework for semi-supervised image classification. Experimental results demonstrate that, while maintaining a comparable classification accuracy, the proposed algorithm performs much more efficient than the state-of-the-art methods.

Manifold regularized extreme learning machine

Extreme learning machine (ELM) works for generalized single-hidden-layer feedforward networks (SLFNs), and its essence is that the hidden layer of SLFNs need not be tuned. But ELM only utilizes labeled data to carry out the supervised learning task. In order to exploit unlabeled data in the ELM model, we first extend the manifold regularization (MR) framework and then demonstrate the relation between the extended MR framework and ELM. Finally, a manifold regularized extreme learning machine is derived from the proposed framework, which maintains the properties of ELM and can be applicable to large-scale learning problems. Experimental results show that the proposed semi-supervised extreme learning machine is the most cost-efficient method. It tends to have better scalability and achieve satisfactory generalization performance at a relatively faster learning speed than traditional semi-supervised learning algorithms.