Linear Learning Methods Research Articles

Dimensionality reduction by LPP‐L21

Locality preserving projection (LPP) is one of the most representative linear manifold learning methods and well exploits intrinsic structure of data. However, the performance of LPP remarkably degenerate in the presence of outliers. To alleviate this problem, the authors propose a robust LPP, namely LPP‐L21. LPP‐L21 employs L2‐norm as the distance metric in spatial dimension of data and L1‐norm as the distance metric over different data points. Moreover, the authors employ L1‐norm to construct similarity graph, this helps to improve robustness of algorithm. Accordingly, the authors present an efficient iterative algorithm to solve LPP‐L21. The authors’ proposed method not only well suppresses outliers but also retains LPP's some nice properties. Experimental results on several image data sets show its advantages.

Towards New Mappings between Emotion Representation Models

There are several models for representing emotions in affect-aware applications, and available emotion recognition solutions provide results using diverse emotion models. As multimodal fusion is beneficial in terms of both accuracy and reliability of emotion recognition, one of the challenges is mapping between the models of affect representation. This paper addresses this issue by: proposing a procedure to elaborate new mappings, recommending a set of metrics for evaluation of the mapping accuracy, and delivering new mapping matrices for estimating the dimensions of a Pleasure-Arousal-Dominance model from Ekman’s six basic emotions. The results are based on an analysis using three datasets that were constructed based on affect-annotated lexicons. The new mappings were obtained with linear regression learning methods. The proposed mappings showed better results on the datasets in comparison with the state-of-the-art matrix. The procedure, as well as the proposed metrics, might be used, not only in evaluation of the mappings between representation models, but also in comparison of emotion recognition and annotation results. Moreover, the datasets are published along with the paper and new mappings might be created and evaluated using the proposed methods. The study results might be interesting for both researchers and developers, who aim to extend their software solutions with affect recognition techniques.

Improved t-SNE based manifold dimensional reduction for remote sensing data processing

In our increasingly “data-abundant” society, remote sensing big data perform massive, high dimension and heterogeneity features, which could result in “dimension disaster” to various extent. It is worth mentioning that the past two decades have witnessed a number of dimensional reductions to weak the spatiotemporal redundancy and simplify the calculation in remote sensing information extraction, such as the linear learning methods or the manifold learning methods. However, the “crowding” and mixing when reducing dimensions of remote sensing categories could degrade the performance of existing techniques. Then in this paper, by analyzing probability distribution of pairwise distances among remote sensing datapoints, we use the 2-mixed Gaussian model(GMM) to improve the effectiveness of the theory of t-Distributed Stochastic Neighbor Embedding (t-SNE). A basic reducing dimensional model is given to test our proposed methods. The experiments show that the new probability distribution capable retains the local structure and significantly reveals differences between categories in a global structure.

Extreme Sparse Multinomial Logistic Regression: A Fast and Robust Framework for Hyperspectral Image Classification

Although sparse multinomial logistic regression (SMLR) has provided a useful tool for sparse classification, it suffers from inefficacy in dealing with high dimensional features and manually set initial regressor values. This has significantly constrained its applications for hyperspectral image (HSI) classification. In order to tackle these two drawbacks, an extreme sparse multinomial logistic regression (ESMLR) is proposed for effective classification of HSI. First, the HSI dataset is projected to a new feature space with randomly generated weight and bias. Second, an optimization model is established by the Lagrange multiplier method and the dual principle to automatically determine a good initial regressor for SMLR via minimizing the training error and the regressor value. Furthermore, the extended multi-attribute profiles (EMAPs) are utilized for extracting both the spectral and spatial features. A combinational linear multiple features learning (MFL) method is proposed to further enhance the features extracted by ESMLR and EMAPs. Finally, the logistic regression via the variable splitting and the augmented Lagrangian (LORSAL) is adopted in the proposed framework for reducing the computational time. Experiments are conducted on two well-known HSI datasets, namely the Indian Pines dataset and the Pavia University dataset, which have shown the fast and robust performance of the proposed ESMLR framework.

Structured Sparse Representation With Union of Data-Driven Linear and Multilinear Subspaces Model for Compressive Video Sampling

The standard compressive sensing (CS) theory can be improved for robust recovery with fewer measurements under the assumption that signals lie on a union of subspaces (UoS). However, the UoS model is restricted to specific types of signal regularities with predetermined topology for subspaces. This paper proposes a generalized model which adaptively decomposes signals into a union of data-driven subspaces (UoDS) for structured sparse representation. The proposed UoDS model leverages subspace clustering to derive the optimal structures and bases for the subspaces conditioned on the sample signals. For multidimensional signals with various statistics, it supports linear and multilinear subspace learning for compressive sampling. As an improvement for generic CS model, the basis which represents the sparsity of sample signals is adaptively generated via linear subspace learning method. Furthermore, a generalized model with multilinear subspace learning is considered for CS to avoid vectorization of high-dimensional signals. In comparison to UoS, the UoDS model requires fewer degrees of freedom for a desirable recovery quality. Experimental results demonstrate that the proposed model for video sampling is promising and applicable.

Document Classification via Nonlinear Metric Learning

Learning a proper distance metric is an important problem in document classification, because the similarities of samples in many problems are usually measured by distance metric. In this paper, we address the nonlinear metric leaning problem with applying in the document classification. First, we propose a new representation about nonlinear metric by using a linear combination of some basic kernels. Second, we give a linear metric learning method by a triplet constraint and k-nearest neighbors, and then we develop it to a nonlinear method based on multiple kernel by above nonlinear metric. Further, the corresponding problem can be rewritten as an unconstrained optimization problem on positive definite matrices groups. At last, to ensure the learned distance matrix must be a positive definite matrix, we provide an improved intrinsic steepest descent algorithm with adaptive step-size to solve this unconstrained optimization. The experimental results show that our proposed method is effective on some document classification problems.

A hybrid intelligent scheme for estimating band gap of doped titanium dioxide semiconductor using crystal lattice distortion

Wide band gap of titanium dioxide (TiO2) semiconductor remains a challenge in photo-catalysis application where light absorption ability of the semiconductor is desired to go beyond ultra-violent region without enhancement of charge recombination rate. In order to address this challenge, two-layer feedforward neural network is proposed for doped titanium dioxide band gap estimation using sensitivity based linear learning method (SBLLM) hybridized with gravitational search algorithm (GSA) for efficient optimization of the number of epoch and hidden neurons. The performance sensitivity of the proposed hybrid SBLLM-GSA model on the gravitational constant and the number of agents are simulated and discussed. The hybrid SBLLM-GSA model outperforms ordinary SBLLM with performance improvement of 13.13% on the basis of root mean square error. The ability of the proposed SBLLM-GSA model to generalize to unseen data was assessed by feeding the model with the crystal lattice parameters of indium doped TiO2, copper-indium co-doped TiO2 as well as sulfur doped TiO2 semiconductor and the obtained band gaps agree well with the measured values. The outcomes of this work indicate the effectiveness of the proposed hybrid model in estimating the band gap of TiO2 semiconductor for efficiency and performance improvement in photo-catalysis application as well as other applications where band gap adjustment is essential for performance enhancement of the semiconductor.

A Comparison of Retweet Prediction Approaches: The Superiority of Random Forest Learning Method

We consider the following retweet prediction task: given a tweet, predict whether it will be retweeted. In the past, a wide range of learning methods and features has been proposed for this task. We provide a systematic comparison of the performance of these learning methods and features in terms of prediction accuracy and feature importance. Specifically, from each previously published approach we take the best performing features and group these into two sets: user features and tweet features. In addition, we contrast five learning methods, both linear and non-linear. On top of that, we examine the added value of a previously proposed time-sensitive modeling approach. To the authors’ knowledge this is the first attempt to collect best performing features and contrast linear and non-linear learning methods. We perform our comparisons on a single dataset and find that user features such as the number of times a user is listed, number of followers, and average number of tweets published per day most strongly contribute to prediction accuracy across selected learning methods. We also find that a random forest-based learning, which has not been employed in previous studies, achieves the highest performance among the learning methods we consider. We also find that on top of properly tuned learning methods the benefits of time-sensitive modeling are very limited.

A quantitative structure–activity relationship to predict efficacy of granular activated carbon adsorption to control emerging contaminants

A quantitative structure–activity relationship was developed to predict the efficacy of carbon adsorption as a control technology for endocrine-disrupting compounds, pharmaceuticals, and components of personal care products, as a tool for water quality professionals to protect public health. Here, we expand previous work to investigate a broad spectrum of molecular descriptors including subdivided surface areas, adjacency and distance matrix descriptors, electrostatic partial charges, potential energy descriptors, conformation-dependent charge descriptors, and Transferable Atom Equivalent (TAE) descriptors that characterize the regional electronic properties of molecules. We compare the efficacy of linear (Partial Least Squares) and non-linear (Support Vector Machine) machine learning methods to describe a broad chemical space and produce a user-friendly model. We employ cross-validation, y-scrambling, and external validation for quality control. The recommended Support Vector Machine model trained on 95 compounds having 23 descriptors offered a good balance between good performance statistics, low error, and low probability of over-fitting while describing a wide range of chemical features. The cross-validated model using a log-uptake (qe) response calculated at an aqueous equilibrium concentration (Ce) of 1 μM described the training dataset with an r2 of 0.932, had a cross-validated r2 of 0.833, and an average residual of 0.14 log units.

Linking demography with drivers: climate and competition

SummaryIn observational demographic data, the number of measured factors that could potentially drive demography (such as daily weather records between two censuses) can easily exceed the number of independent observations. Thus, identifying the important drivers requires alternatives to standard model selection and variable selection methods.Spline methods that estimate smooth functions over continuous domains (such as space or time) have the potential to resolve high‐dimensional problems in ecological systems. We consider two examples that are important for many plant populations: competition with neighbours that vary in size and distance from the focal individual and climate variables during a window of time before a response (growth, survival, etc.) is measured.For competition covariates, we use a simulation study based on empirical data to show that a monotone spline estimate of competition kernels via approximate AIC returns very accurate estimates. We then apply the method to long‐term, mapped quadrat data on the four dominant species in an Idaho (US) sagebrush steppe community.For climate predictors and their temporal lags, we use simulated data sets to compare functional smoothing methods with competing linear (LASSO) or machine learning (random forests) methods. Given sufficient data, functional smoothing methods outperformed the other two methods.Functional smoothing methods can advance data‐driven population modelling by providing alternatives to specifying competition kernelsa prioriand to arbitrarily aggregating continuous environmental covariates. However, there are important open questions related to modelling of nonlinear climate responses and size × climate interactions.

Multi-class Fukunaga Koontz discriminant analysis for enhanced face recognition

Linear subspace learning methods such as Fisher׳s Linear Discriminant Analysis (LDA), Unsupervised Discriminant Projection (UDP), and Locality Preserving Projections (LPP) have been widely used in face recognition applications as a tool to capture low dimensional discriminant information. However, when these methods are applied in the context of face recognition, they often encounter the small-sample-size problem. In order to overcome this problem, a separate Principal Component Analysis (PCA) step is usually adopted to reduce the dimensionality of the data. However, such a step may discard dimensions that contain important discriminative information that can aid classification performance. In this work, we propose a new idea which we named Multi-class Fukunaga Koontz Discriminant Analysis (FKDA) by incorporating the Fukunaga Koontz Transform within the optimization for maximizing class separation criteria in LDA, UDP, and LPP. In contrast to traditional LDA, UDP, and LPP, our approach can work with very high dimensional data as input, without requiring a separate dimensionality reduction step to make the scatter matrices full rank. In addition, the FKDA formulation seeks optimal projection direction vectors that are orthogonal which the existing methods cannot guarantee, and it has the capability of finding the exact solutions to the “trace ratio” objective in discriminant analysis problems while traditional methods can only deal with a relaxed and inexact “ratio trace” objective. We have shown using six face database, in the context of large scale unconstrained face recognition, face recognition with occlusions, and illumination invariant face recognition, under “closed set”, “semi-open set”, and “open set” recognition scenarios, that our proposed FKDA significantly outperforms traditional linear discriminant subspace learning methods as well as five other competing algorithms.

Shape recognition using orientational and morphological scale‐spaces of curvatures

In this study, a scale‐invariant representation for closed planar curves (silhouettes) is proposed. The orientations of all points within the Gaussian scale‐space of the curve are extracted. This orientation scale‐space is used to create the silhouette orientation image in which the positions of each pixel indicate the curve's pixel positions and scales, whereas the colour represents orientation. The representation is extracted for multiple levels of the morphological scale‐space of the silhouette. The proposed representation is invariant to scale and transformable under planar rotation. Using linear and non‐linear distance learning methods, experiments on the MPEG7, ETH80 and Kimia shape datasets were conducted, with results indicating an advanced recognition capability.

Estimation of surface tension of methyl esters biodiesels using computational intelligence technique

Due to environmental benefits, methyl esters biodiesel got a considerable attention as a viable substitute to petroleum-based diesel. Surface tension plays significant role in atomization of this biodiesel since it controls the combustion process inside the engine through fuel–air mixing. Experimental determination of the surface tension of biodiesel is expensive and time consuming which limits its application as substitute for petroleum-based diesel. This is because proper choice of any methyl esters for diesel engine applications depend on the value of surface tension as high value of surface tension brings about difficulty in droplet formation. This work employs computational intelligence technique on the platform of sensitivity based linear learning method (SBLLM) to develop methyl esters surface tension estimator (MESTE) which estimates surface tension of methyl esters biodiesel with high degree of accuracy. Surface tensions of eight different classes of methyl esters were estimated at different temperatures by training and testing of neural network using SBLLM. The estimated surface tensions were compared with experimental results as well as surface tension obtained from Parachor model and Goldhammer model. The outstanding performance of the developed MESTE suggests its potential in estimating surface tension of methyl esters biodiesel for enhancing the atomization in biodiesels engine applications.

Nonlinear Metric Learning with Kernel Density Estimation

Metric learning, the task of learning a good distance metric, is a key problem in machine learning with ample applications. This paper introduces a novel framework for nonlinear metric learning, called kernel density metric learning (KDML), which is easy to use and provides nonlinear, probability-based distance measures. KDML constructs a direct nonlinear mapping from the original input space into a feature space based on kernel density estimation. The nonlinear mapping in KDML embodies established distance measures between probability density functions, and leads to accurate classification on datasets for which existing linear metric learning methods would fail. It addresses the severe challenge to distance-based classifiers when features are from heterogeneous domains and, as a result, the Euclidean or Mahalanobis distance between original feature vectors is not meaningful. We also propose two ways to determine the kernel bandwidths, including an adaptive local scaling approach and an integrated optimization algorithm that learns the Mahalanobis matrix and kernel bandwidths together. KDML is a general framework that can be combined with any existing metric learning algorithm. As concrete examples, we combine KDML with two leading metric learning algorithms, large margin nearest neighbors (LMNN) and neighborhood component analysis (NCA). KDML can naturally handle not only numerical features, but also categorical ones, which is rarely found in previous metric learning algorithms. Extensive experimental results on various datasets show that KDML significantly improves existing metric learning algorithms in terms of classification accuracy.

Machine learning approaches for integrating clinical and imaging features in late-life depression classification and response prediction.

Currently, depression diagnosis relies primarily on behavioral symptoms and signs, and treatment is guided by trial and error instead of evaluating associated underlying brain characteristics. Unlike past studies, we attempted to estimate accurate prediction models for late-life depression diagnosis and treatment response using multiple machine learning methods with inputs of multi-modal imaging and non-imaging whole brain and network-based features. Late-life depression patients (medicated post-recruitment) (n = 33) and older non-depressed individuals (n = 35) were recruited. Their demographics and cognitive ability scores were recorded, and brain characteristics were acquired using multi-modal magnetic resonance imaging pretreatment. Linear and nonlinear learning methods were tested for estimating accurate prediction models. A learning method called alternating decision trees estimated the most accurate prediction models for late-life depression diagnosis (87.27% accuracy) and treatment response (89.47% accuracy). The diagnosis model included measures of age, Mini-mental state examination score, and structural imaging (e.g. whole brain atrophy and global white mater hyperintensity burden). The treatment response model included measures of structural and functional connectivity. Combinations of multi-modal imaging and/or non-imaging measures may help better predict late-life depression diagnosis and treatment response. As a preliminary observation, we speculate that the results may also suggest that different underlying brain characteristics defined by multi-modal imaging measures-rather than region-based differences-are associated with depression versus depression recovery because to our knowledge this is the first depression study to accurately predict both using the same approach. These findings may help better understand late-life depression and identify preliminary steps toward personalized late-life depression treatment.

Modeling permeability and PVT properties of oil and gas reservoir using hybrid model based on type-2 fuzzy logic systems

In this work, the power of type-2 fuzzy logic system is demonstrated by using it to improve the prediction of permeability and PVT properties in a hybrid model setup. Hybrid intelligent model through the fusion of type-2 FLS (T2) and sensitivity-based linear learning method (SBLLM) is presented, and is hereby referred to as T2-SBLLM hybrid model. SBLLM, as a learning tool, has gained popularity due to its unique characteristics and performance. However, the generalization capability of SBLLM and other neural network-based solutions often depends on the nature of the dataset, particularly on whether uncertainty is present in the dataset or not. This work proposes a hybrid system through a combination of type-2 fuzzy logic systems (type-2 FLS) and SBLLM, and then uses it to model both permeability and PVT properties of oil and gas reservoir; type-2 FLS has been chosen to be a precursor to SBLLM in order to better handle uncertainties existing in the datasets. The type-2 FLS is used to first handle uncertainties in the reservoir data so that the final output is then passed to the SBLLM for training and then final prediction is done using the unseen testing dataset. Comparative studies have been carried out using different industrial reservoir data for both permeability and PVT properties. Empirical results show that the proposed T2-SBLLM hybrid system outperformed each of the type-2 FLS and SBLLM, as the two constituent models, in all cases, with the improvement made to the SBLLM performance being far higher compared to that of type-2 FLS, since type-2 FLS is originally adept at modeling uncertainties.

Linear vs non-linear learning methods A comparative study for forest above ground biomass, estimation from texture analysis of satellite images

The aboveground biomass estimation is an important question in the scope of Reducing Emission from Deforestation and Forest Degradation (REDD framework of the UNCCC). It is particularly challenging for tropical countries because of the scarcity of accurate ground forest inventory data and of the complexity of the forests. Satellite-borne remote sensing can help solve this problem considering the increasing availability of optical very high spatial resolution images that provide information on the forest structure via texture analysis of the canopy grain. For example, the FOTO (FOurier Texture Ordination) proved relevant for forest biomass prediction in several tropical regions. It uses PCA and linear regression and, in this paper, we suggest applying classification methods such as k-NN (k-nearest neighbors), SVM (support vector machines) and Random Forests to texture descriptors extracted from images via Fourier spectra. Experiments have been carried out on simulated images produced by the software DART (Discrete Anisotropic Radiative Transfer) in reference to information (3D stand mockups) from forests of DRC (Democratic Republic of Congo), CAR (Central African Republic) and Congo. On this basis, we show that some classification techniques may yield a gain in prediction accuracy of 18 to 20%

Dynamic adjustment of hidden node parameters for extreme learning machine.

Extreme learning machine (ELM), proposed by Huang et al., was developed for generalized single hidden layer feedforward networks with a wide variety of hidden nodes. ELMs have been proved very fast and effective especially for solving function approximation problems with a predetermined network structure. However, it may contain insignificant hidden nodes. In this paper, we propose dynamic adjustment ELM (DA-ELM) that can further tune the input parameters of insignificant hidden nodes in order to reduce the residual error. It is proved in this paper that the energy error can be effectively reduced by applying recursive expectation-minimization theorem. In DA-ELM, the input parameters of insignificant hidden node are updated in the decreasing direction of the energy error in each step. The detailed theoretical foundation of DA-ELM is presented in this paper. Experimental results show that the proposed DA-ELM is more efficient than the state-of-art algorithms such as Bayesian ELM, optimally-pruned ELM, two-stage ELM, Levenberg-Marquardt, sensitivity-based linear learning method as well as the preliminary ELM.

Statistical and Machine Learning Methods for Software Fault Prediction Using CK Metric Suite: A Comparative Analysis

Experimental validation of software metrics in fault prediction for object-oriented methods using statistical and machine learning methods is necessary. By the process of validation the quality of software product in a software organization is ensured. Object-oriented metrics play a crucial role in predicting faults. This paper examines the application of linear regression, logistic regression, and artificial neural network methods for software fault prediction using Chidamber and Kemerer (CK) metrics. Here, fault is considered as dependent variable and CK metric suite as independent variables. Statistical methods such as linear regression, logistic regression, and machine learning methods such as neural network (and its different forms) are being applied for detecting faults associated with the classes. The comparison approach was applied for a case study, that is, Apache integration framework (AIF) version 1.6. The analysis highlights the significance of weighted method per class (WMC) metric for fault classification, and also the analysis shows that the hybrid approach of radial basis function network obtained better fault prediction rate when compared with other three neural network models.

Modelling and predicting the biological effects of nanomaterials

The commercial applications of nanoparticles are growing rapidly, but we know relatively little about how nanoparticles interact with biological systems. Their value – but also their risk – is related to their nanophase properties being markedly different to those of the same material in bulk. Experiments to determine how nanoparticles are taken up, distributed, modified, and elicit any adverse effects are essential. However, cost and time considerations mean that predictive models would also be extremely valuable, particularly assisting regulators to minimize health and environmental risks.We used novel sparse machine learning methods that employ Bayesian neural networks to model three nanoparticle data sets using both linear and nonlinear machine learning methods. The first data comprised iron oxide nanoparticles decorated with 108 different molecules tested against five cell lines, HUVEC, pancreatic cancer, and three macrophage or macrophage-like lines. The second data set comprised 52 nanoparticles with various core compositions, coatings, and surface attachments. The nanoparticles were characterized using four descriptors (size, relaxivities, and zeta potential), and their biological effects on four cells lines assessed using four biological assays per cell line and four concentrations per assay. The third data set involved the biological responses to gold nanoparticles functionalized by 80 different small molecules. Nonspecific binding and binding to AChE were the biological endpoints modelled. The biological effects of nanoparticles were modelled using molecular descriptors for the molecules that decorated the nanoparticle surface. Models with good statistical quality were constructed for most biological endpoints. These proof-of-concept models show that modelling biological effects of nanomaterials is possible using modern modelling methods.