Low Generalization Error Research Articles

Comparison of hyperspectral and multi-spectral imagery to building a spectral library and land cover classification performance

The main aim of this research work is to compare k -nearest neighbor algorithm (KNN) supervised classification with migrating means clustering unsupervised classification (MMC) method on the performance of hyperspectral and multispectral data for spectral land cover classes and develop their spectral library in Samara, Russia. Accuracy assessment of the derived thematic maps was based on the analysis of the classification confusion matrix statistics computed for each classified map, using for consistency the same set of validation points. We were analyzed and compared Earth Observing-1 (EO-1) Hyperion hyperspectral data to Landsat 8 Operational Land Imager (OLI) and Advance Land Imager (ALI) multispectral data. Hyperspectral imagers, currently available on airborne platforms, provide increased spectral resolution over existing space based sensors that can document detailed information on the distribution of land cover classes, sometimes species level. Results indicate that KNN (95, 94, 88 overall accuracy and .91, .89, .85 kappa coefficient for Hyp, ALI, OLI respectively) shows better results than unsupervised classification (93, 90, 84 overall accuracy and .89, .87, .81 kappa coefficient for Hyp, ALI, OLI respectively). Development of spectral library for land cover classes is a key component needed to facilitate advance analytical techniques to monitor land cover changes. Different land cover classes in Samara were sampled to create a common spectral library for mapping landscape from remotely sensed data. The development of these libraries provides a physical basis for interpretation that is less subject to conditions of specific data sets, to facilitate a global approach to the application of hyperspectral imagers to mapping landscape. In addition, it is demonstrated that the hyperspectral satellite image provides more accurate classification results than those extracted from the multispectral satellite image. The higher classification accuracy by KNN supervised was attributed principally to the ability of this classifier to identify optimal separating classes with low generalization error, thus producing the best possible classes’ separation.

Radiology report annotation using intelligent word embeddings: Applied to multi-institutional chest CT cohort

We proposed an unsupervised hybrid method – Intelligent Word Embedding (IWE) that combines neural embedding method with a semantic dictionary mapping technique for creating a dense vector representation of unstructured radiology reports. We applied IWE to generate embedding of chest CT radiology reports from two healthcare organizations and utilized the vector representations to semi-automate report categorization based on clinically relevant categorization related to the diagnosis of pulmonary embolism (PE). We benchmark the performance against a state-of-the-art rule-based tool, PeFinder and out-of-the-box word2vec. On the Stanford test set, the IWE model achieved average F1 score 0.97, whereas the PeFinder scored 0.9 and the original word2vec scored 0.94. On UPMC dataset, the IWE model’s average F1 score was 0.94, when the PeFinder scored 0.92 and word2vec scored 0.85. The IWE model had lowest generalization error with highest F1 scores. Of particular interest, the IWE model (trained on the Stanford dataset) outperformed PeFinder on the UPMC dataset which was used originally to tailor the PeFinder model.

Machine learning applied to estimating broadband source signature characteristics in a shallow ocean environment

Probabilistic machine and deep learning methods are critical elements in the area of automation and artificial intelligence. The regression task considered here is to use supervised machine learning to predict a frequency dependent acoustic source level from received broadband signals that have propagated through a shallow ocean waveguide possessing random features. The details are intimately linked to a previously introduced sequential maximum entropy—Bayesian approach employed to generate marginal probability distributions for both environmental and source parameter values. To meet the requirement to have both low training and generalization errors, several regularization and non-linear optimization methods are considered to enhance performance. This includes convolutional networks. Optimization includes not only the issue of sampling but also finding an effective model capacity, which is one of the most significant challenges to successful machine learning applications. The methodology is applied to measured broadband data collected in about 75 m of water about 60 miles south of Cape Cod Massachusetts in an area called the mud pond. Both the training samples and the samples not used in training have known source levels with which to measure both the training error and the generalization error and thus quantify performance.

Assessing Generalization Ability of Majority Vote Point Classifiers.

Classification algorithms have been traditionally designed to simultaneously reduce errors caused by bias as well by variance. However, there occur many situations where low generalization error becomes extremely crucial to getting tangible classification solutions, and even slight overfitting causes serious consequences in the test results. In such situations, classifiers with low Vapnik-Chervonenkis (VC) dimension can bring out positive differences due to two main advantages: 1) the classifier manages to keep the test error close to training error and 2) the classifier learns effectively with small number of samples. This paper shows that a class of classifiers named majority vote point (MVP) classifiers, on account of very low VC dimension, can exhibit a generalization error that is even lower than that of linear classifiers. This paper proceeds by theoretically formulating an upper bound on the VC dimension of the MVP classifier. Later, through empirical analysis, the trend of exact values of VC dimension is estimated. Finally, case studies on machine fault diagnosis problems and prostate tumor detection problem revalidate the fact that an MVP classifier can achieve a lower generalization error than most other classifiers.

Multiple empirical kernel learning with locality preserving constraint

Multiple Kernel Learning (MKL) is flexible in dealing with problems involving multiple and heterogeneous data sources. However, the necessity of inner-product form restricts its application since to kernelize the algorithms unsatisfying the inner-product formulation is pretty difficult. To overcome this problem, Multiple Empirical Kernel Learning (MEKL) is proposed by explicitly mapping input samples to feature spaces, in which the mapped feature vectors are explicitly presented. Most existed MEKLs optimize the learning framework by minimizing empirical risk, regularization risk and the loss term of multiple feature spaces. As little attention is paid to preserving local structure among training samples, the learned classifier might lack of locality similarity preserving property, which might result in unfavorable performance. Inspired by Locality Preserving Projection (LPP) which is to seek the optimal projection by preserving the local property of input samples, we introduce the locality preserving constraint into the learning framework to propose a novel Multiple Empirical Kernel Learning with Locality Preserving Constraint (MEKL-LPC). MEKL-LPC shows lower generalization error bound than both the Modification of Ho–Kashyap algorithm with Squared approximation of the misclassification error (MHKS) and Multi-Kernel MHKS (MultiK-MHKS) in terms of Rademacher complexity. Experiments on several real-world datasets demonstrate that MEKL-LPC outperforms the compared algorithms. The contributions of this work are: (i) originally integrating locality preserving constraint into MEKL, (ii) proposing a lower generalization error bound algorithm, i.e. MEKL-LPC.

υ-Support vector machine based on discriminant sparse neighborhood preserving embedding

In this paper, we mainly focus on two issues (1) SVM is very sensitive to noise. (2) The solution of SVM does not take into consideration of the intrinsic structure and the discriminant information of the data. To address these two problems, we first propose an integration model to integrate both the local manifold structure and the local discriminant information into l1 graph embedding. Then we add the integration model into the objection function of υ-support vector machine. Therefore, a discriminant sparse neighborhood preserving embedding υ-support vector machine (υ-DSNPESVM) method is proposed. The theoretical analysis demonstrates that υ-DSNPESVM is a reasonable maximum margin classifier and can obtain a very lower generalization error upper bound by minimizing the integration model and the upper bound of margin error. Moreover, in the nonlinear case, we construct the kernel sparse representation-based l1 graph for υ-DSNPESVM, which is more conducive to improve the classification accuracy than l1 graph constructed in the original space. Experimental results on real datasets show the effectiveness of the proposed υ-DSNPESVM method.

Face Recognition across Time Lapse Using Convolutional Neural Networks

Time lapse, characteristic of aging, is a complex process that affects the reliability and security of biometric face recognition systems. This paper reports the novel use and effectiveness of deep learning, in general, and convolutional neural networks (CNN), in particular, for automatic rather than hand-crafted feature extraction for robust face recognition across time lapse. A CNN architecture using the VGG-Face deep (neural network) learning is found to produce highly discriminative and interoperable features that are robust to aging variations even across a mix of biometric datasets. The features extracted show high inter-class and low intra-class variability leading to low generalization errors on aging datasets using ensembles of subspace discriminant classifiers. The classification results for the all-encompassing authentication methods proposed on the challenging FG-NET and MORPH datasets are competitive with state-of-the-art methods including commercial face recognition engines and are richer in functionality and interoperability than existing methods as it handles mixed biometric datasets, e.g., FG-NET and MORPH.

Ensemble based point and confidence interval forecasting in software engineering

In this paper, we propose and use analytical and heuristic approaches for forecasting 95% confidence intervals (CIs) for software effort or size parameter. The analytical and heuristics forecasts are combined to create an ensemble to lower generalization error. Using ordinary least squares (OLS) and Bayesian regression models, we create three different ensembles. These three different ensembles are one maximizing maximum likelihood hypothesis (MLE), one maximizing maximum a posteriori hypothesis (MAP), a hybrid one maximizing both MLE and MAP hypotheses. Using two different performance metrics, we test the three different ensembles on real-world software engineering datasets. The results of our experiments indicated that the hybrid ensemble shows the worst performance while the MLE ensemble provides superior performance in most cases.

Noise peeling methods to improve boosting algorithms

Boosting refers to a family of methods that combine sequences of individual classifiers into highly accurate ensemble models through weighted voting. AdaBoost, short for “Adaptive Boosting”, is the most well-known boosting algorithm. AdaBoost has many strengths. Among them, there is sufficient empirical evidence pointing to its performance being generally superior to that of individual classifiers. In addition, even when combining a large number of weak learners, AdaBoost can be very robust to overfitting usually with lower generalization error than other competing ensemble methodologies, such as bagging and random forests. However, AdaBoost, as most hard margin classifiers, tends to be sensitive to outliers and noisy data, since it assigns observations that have been misclassified a higher weight in subsequent iterations. It has recently been proven that for any booster with a potential convex loss function, and any nonzero random classification noise rate, there is a data set, which can be efficiently learnable by the booster if there is no noise, but cannot be learned with accuracy better than 1/2 with random classification noise present. Several techniques to identify and potentially delete (peel) noisy samples in binary classification are proposed in order to improve the performance of AdaBoost. It is found that peeling methods generally perform better than AdaBoost and other noise resistant boosters, especially when high levels of noise are present in the data.

Disjunctive normal random forests

We develop a novel supervised learning/classification method, called disjunctive normal random forest (DNRF). A DNRF is an ensemble of randomly trained disjunctive normal decision trees (DNDT). To construct a DNDT, we formulate each decision tree in the random forest as a disjunction of rules, which are conjunctions of Boolean functions. We then approximate this disjunction of conjunctions with a differentiable function and approach the learning process as a risk minimization problem that incorporates the classification error into a single global objective function. The minimization problem is solved using gradient descent. DNRFs are able to learn complex decision boundaries and achieve low generalization error. We present experimental results demonstrating the improved performance of DNDTs and DNRFs over conventional decision trees and random forests. We also show the superior performance of DNRFs over state-of-the-art classification methods on benchmark datasets.

Ultra-Short-Term Wind Speed and Power Forecast Based on Dynamic Selective Neural Network Ensemble

With the scale of grid-connected wind farms increasing, accurate forecast of ultra-short-term wind speed and wind power is very important to the stable operation of power systems. This paper presents a dynamic selective neural network ensemble (DSNNE) forecast method, which makes use of K nearest neighbor algorithm to collect the generalization errors of certain different BP neural networks and RBF neural networks into a performance matrix and then the neural networks with low local generalization errors are dynamically selected and locally dynamic averaging is applied to the neural networks in order to conduct the final results of the ensemble. Then this method is applied to realize the wind speed and power ultra-short-term advance forecast, taking the wind speed and wind turbine power output from a wind farm in China as the original data. The research results show that DSNNE improves the generalization ability of the neural network system and the prediction accuracy of wind power and wind speed significantly. It proves the validity and effectiveness of the DSNNE with controlling the biggest mean relative error of 2 minutes ahead wind power and wind speed forecast as low as 25% and 16% respectively.

Sparse pose manifolds

The efficient manipulation of randomly placed objects relies on the accurate estimation of their 6 DoF geometrical configuration. In this paper we tackle this issue by following the intuitive idea that different objects, viewed from the same perspective, should share identical poses and, moreover, these should be efficiently projected onto a well-defined and highly distinguishable subspace. This hypothesis is formulated here by the introduction of pose manifolds relying on a bunch-based structure that incorporates unsupervised clustering of the abstracted visual cues and encapsulates appearance and geometrical properties of the objects. The resulting pose manifolds represent the displacements among any of the extracted bunch points and the two foci of an ellipse fitted over the members of the bunch-based structure. We post-process the established pose manifolds via $$l_1$$ l 1 norm minimization so as to build sparse and highly representative input vectors that are characterized by large discrimination capabilities. While other approaches for robot grasping build high dimensional input vectors, thus increasing the complexity of the system, in contrast, our method establishes highly distinguishable manifolds of low dimensionality. This paper represents the first integrated research endeavor in formulating sparse pose manifolds, with experimental results providing evidence of low generalization error, justifying thus our theoretical claims.

Honey characterization using computer vision system and artificial neural networks

This paper reports the development of a computer vision system (CVS) for non-destructive characterization of honey based on colour and its correlated chemical attributes including ash content (AC), antioxidant activity (AA), and total phenolic content (TPC). Artificial neural network (ANN) models were applied to transform RGB values of images to CIE L∗a∗b∗ colourimetric measurements and to predict AC, TPC and AA from colour features of images. The developed ANN models were able to convert RGB values to CIE L∗a∗b∗ colourimetric parameters with low generalization error of 1.01±0.99. In addition, the developed models for prediction of AC, TPC and AA showed high performance based on colour parameters of honey images, as the R2 values for prediction were 0.99, 0.98, and 0.87, for AC, AA and TPC, respectively. The experimental results show the effectiveness and possibility of applying CVS for non-destructive honey characterization by the industry.

The role of margins in boosting and ensemble performance

Boosting refers to methods that create a sequence of classifiers that perform at least slightly better than random (weak learners) and combine them into a highly accurate ensemble model (strong learners) through weighted voting. There is sufficient empirical evidence to suggest that the performance of boosting methods is superior to that of individual classifiers. In the bias–variance decomposition framework, it has been demonstrated that boosting algorithms typically reduce bias for learning problems, and in some instances reduce the variance. In addition, even when combining a large number of weak learners, boosting algorithms can be very robust to overfitting, in most instances having lower generalization error than other competing ensemble methodologies, such as bagging and random forests. To explain the successful performance of boosting methods, Schapire et al. (Boosting the margin: a new explanation for the effectiveness of voting methods. Ann Stat 1998, 26:1651–1686) developed a bound based on the margins of the training data, from which they concluded that larger margins should lead to lower generalization error. In this article, we examine the role of margins in boosting and ensemble method performance. WIREs Comput Stat 2014, 6:124–131. doi: 10.1002/wics.1292This article is categorized under: Statistical and Graphical Methods of Data Analysis > Bootstrap and Resampling Statistical Learning and Exploratory Methods of the Data Sciences > Classification and Regression Trees (CART)

On Generalizable Low False-Positive Learning Using Asymmetric Support Vector Machines

The Support Vector Machines (SVMs) have been widely used for classification due to its ability to give low generalization error. In many practical applications of classification, however, the wrong prediction of a certain class is much severer than that of the other classes, making the original SVM unsatisfactory. In this paper, we propose the notion of Asymmetric Support Vector Machine (ASVM), an asymmetric extension of the SVM, for these applications. Different from the existing SVM extensions such as thresholding and parameter tuning, ASVM employs a new objective that models the imbalance between the costs of false predictions from different classes in a novel way such that user tolerance on false-positive rate can be explicitly specified. Such a new objective formulation allows us of obtaining a lower false-positive rate without much degradation of the prediction accuracy or increase in training time. Furthermore, we show that the generalization ability is preserved with the new objective. We also study the effects of the parameters in ASVM objective and address some implementation issues related to the Sequential Minimal Optimization (SMO) to cope with large-scale data. An extensive simulation is conducted and shows that ASVM is able to yield either noticeable improvement in performance or reduction in training time as compared to the previous arts.

Efficient representation and feature extraction for neural network-based 3D object pose estimation

This paper introduces an efficient representation and feature extraction technique for 3D pose estimation of objects, incorporating a novel mechanism for the exploitation of the extracted visual cues. A combination of a fuzzy clustering technique for the input space, with supervised learning, results in a problem of reduced dimensionality and an efficient mapping of the input–output space. While other neural network-based approaches for 3D pose estimation focus on reducing dimensionality based on input space characteristics, such as with PCA-based approaches, the proposed scheme directly targets the input–output mapping, based on the available visual data. Evaluation results provide evidence of low generalization error when estimating the 3D pose of objects, with the best performance achieved when employing Radial Basis Functions. The proposed system can be adopted in several computer vision applications requiring object localization, pose estimation and target tracking.

ARTIFICAL NEURAL NETWORKS: The New Era Of I.T

Artificial neural networks are composed of interconnecting artificial neurons (programming constructs that mimic the properties of biological neurons). Artificial neural networks may either be used to gain an understanding of biological neural networks, or for solving artificial intelligence problems without necessarily creating a model of a real biological system. The real, biological nervous system is highly complex: artificial neural network algorithms attempt to abstract this complexity and focus on what may hypothetically matter most from an information processing point of view. Good performance (e.g. as measured by good predictive ability, low generalization error), or performance mimicking animal or human error patterns, can then be used as one source of evidence towards supporting the hypothesis that the abstraction really captured something important from the point of view of information processing in the brain. Another incentive for these abstractions is to reduce the amount of computation required to simulate artificial neural networks, so as to allow one to experiment with larger networks and train them on larger data sets. The key element of this paradigm is the novel structure of the information processing system. It is composed of a large number of highly interconnected processing elements (neurones) working in unison to solve specific problems. ANNs, like people, learn by example. An ANN is configured for a specific application, such as pattern recognition or data classification, through a learning process. Learning in biological systems involves adjustments to the synaptic connections that exist between the neurones. This is true of ANNs as well

Dynamic fusion method using Localized Generalization Error Model

Multiple Classifier Systems (MCSs), which combine the outputs of a set of base classifiers, were proposed as a method to develop a more accurate classification system. One fundamental issue is how to combine the base classifiers. In this paper, a new dynamic fusion method named Localized Generalization Error Model Fusion Method (LFM) for MCSs is proposed. The Localized Generalization Error Model (L-GEM) has been used to estimate the local competence of base classifiers in MCSs. L-GEM provides a generalization error bound for unseen samples located within neighborhoods of testing samples. Base classifiers with lower generalization error bounds are assigned higher weights. In contrast to the current dynamic fusion methods, LFM estimates the local competence of base classifiers not only using the information of training error but also the sensitivity of classifier outputs. The additional effect of the sensitivity on the performance of model and the time complexity of the LFM are discussed and analyzed. Experimental results show that the MCSs using the LFM as a combination method outperform those using the other 21 dynamic fusion methods in terms of testing accuracy and time.

Efficient feature selection filters for high-dimensional data

Feature selection is a central problem in machine learning and pattern recognition. On large datasets (in terms of dimension and/or number of instances), using search-based or wrapper techniques can be computationally prohibitive. Moreover, many filter methods based on relevance/redundancy assessment also take a prohibitively long time on high-dimensional datasets.In this paper, we propose efficient unsupervised and supervised feature selection/ranking filters for high-dimensional datasets. These methods use low-complexity relevance and redundancy criteria, applicable to supervised, semi-supervised, and unsupervised learning, being able to act as pre-processors for computationally intensive methods to focus their attention on smaller subsets of promising features. The experimental results, with up to 105 features, show the time efficiency of our methods, with lower generalization error than state-of-the-art techniques, while being dramatically simpler and faster.

Hyperion hyperspectral imagery analysis combined with machine learning classifiers for land use/cover mapping

Describing the pattern and the spatial distribution of land cover is traditionally based on remote sensing data analysis and one of the most commonly techniques applied has been image classification. The main objective of the present study has been to evaluate the combined use of Hyperion hyperspectral imagery with the Support Vector Machines (SVMs) and Artificial Neural Networks (ANNs) classifiers for discriminating land-cover classes in a typical Mediterranean setting. Accuracy assessment of the derived thematic maps was based on the analysis of the classification confusion matrix statistics computed for each classification map, using for consistency the same set of validation points.Results indicated a close classification accuracy between the two classifiers, with the SVMs somehow outperforming the ANNs by 3.31% overall accuracy and by 0.038 kappa coefficient. Although both classifiers produced close results, SVMs generally appeared most useful in describing the spatial distribution and the cover density of each land cover category. The higher classification accuracy by SVMs was attributed principally to the ability of this classifier to identify an optimal separating hyperplane for classes’ separation which allows a low generalization error, thus producing the best possible classes’ separation. On the other, as a key disadvantage of both techniques was identified that both do not operate on a sub-pixel level, which can significantly reduce their accuracy due to possible mixture problems occurred when coarse spatial resolution remote sensing imagery is used.All in all, this study demonstrated that, provided that a Hyperion hyperspectral imagery can be made available at regular time intervals over a given region, when combined with either SVMs or ANNs classifiers, can potentially enable a wider approach in land use/cover mapping. This can be of particular importance, especially for regions like in the Mediterranean basin, since it can be related to mapping and monitoring of land degradation and desertification phenomena which are evident in such areas.