Superior Generalization Capability Research Articles

Hybrid Support Vector Regression with Parallel Co-Evolution Algorithm Based on GA and PSO for Forecasting Monthly Rainfall

Accurate and timely monthly rainfall forecasting is a major challenge for the scientific community in hydrological research such as river management project and design of flood warning systems. Support Vector Regression (SVR) is a very useful precipitation prediction model. In this paper, a novel parallel co-evolution algorithm is presented to determine the appropriate parameters of the SVR in rainfall prediction based on parallel co-evolution by hybrid Genetic Algorithm and Particle Swarm Optimization algorithm, namely SVRGAPSO, for monthly rainfall prediction. The framework of the parallel co-evolutionary algorithm is to iterate two GA and PSO populations simultaneously, which is a mechanism for information exchange between GA and PSO populations to overcome premature local optimum. Our methodology adopts a hybrid PSO and GA for the optimal parameters of SVR by parallel co-evolving. The proposed technique is applied over rainfall forecasting to test its generalization capability as well as to make comparative evaluations with the several competing techniques, such as the other alternative methods, namely SVRPSO (SVR with PSO), SVRGA (SVR with GA), and SVR model. The empirical results indicate that the SVRGAPSO results have a superior generalization capability with the lowest prediction error values in rainfall forecasting. The SVRGAPSO can significantly improve the rainfall forecasting accuracy. Therefore, the SVRGAPSO model is a promising alternative for rainfall forecasting.

Imaging through glass diffusers using densely connected convolutional networks

Computational imaging through scatter generally is accomplished by first characterizing the scattering medium so that its forward operator is obtained and then imposing additional priors in the form of regularizers on the reconstruction functional to improve the condition of the originally ill-posed inverse problem. In the functional, the forward operator and regularizer must be entered explicitly or parametrically (e.g., scattering matrices and dictionaries, respectively). However, the process of determining these representations is often incomplete, prone to errors, or infeasible. Recently, deep learning architectures have been proposed to instead learn both the forward operator and regularizer through examples. Here, we propose for the first time, to our knowledge, a convolutional neural network architecture called “IDiffNet” for the problem of imaging through diffuse media and demonstrate that IDiffNet has superior generalization capability through extensive tests with well-calibrated diffusers. We also introduce the negative Pearson correlation coefficient (NPCC) loss function for neural net training and show that the NPCC is more appropriate for spatially sparse objects and strong scattering conditions. Our results show that the convolutional architecture is robust to the choice of prior, as demonstrated by the use of multiple training and testing object databases, and capable of achieving higher space–bandwidth product reconstructions than previously reported.

Deep Convolutional Network Based on Pyramid Architecture

Deep convolutional network demonstrates that the classification accuracy can be remarkably improved by increasing the number of network layers, however, increases the accuracy by 1% of costs nearly doubling the number of layers. Meanwhile, gradient dispersion will occur in the training process, which leads to performance degradation. In order to solve the problem of training difficulty with the increased number of layers, we focus on network architecture and propose a deep convolutional network based on the pyramid structure. In the network architecture, as the number of layers increased, the feature map dimensions (i.e., the number of channels) are gradually increased at each layer to distribute the burden concentrated at locations of structural units affected by downsampling, such that all units are equally distributed. By exploring the sequence between the stacked elements inside the structural unit, we design a pyramidal building block, as its shape gradually widens from the top downwards, which is called the deep pyramid convolutional network (DPCNet). Experimental results on CIFAR-10 and CIFAR-100 datasets have shown that DPCNet has the superior generalization capability and can effectively improve the image classification accuracy.

Considering diversity and accuracy simultaneously for ensemble pruning

Diversity among individual classifiers is widely recognized to be a key factor to successful ensemble selection, while the ultimate goal of ensemble pruning is to improve its predictive accuracy. Diversity and accuracy are two important properties of an ensemble. Existing ensemble pruning methods always consider diversity and accuracy separately. However, in contrast, the two closely interrelate with each other, and should be considered simultaneously. Accordingly, three new measures, i.e., Simultaneous Diversity & Accuracy, Diversity-Focused-Two and Accuracy-Reinforcement, are developed for pruning the ensemble by greedy algorithm. The motivation for Simultaneous Diversity & Accuracy is to consider the difference between the subensemble and the candidate classifier, and simultaneously, to consider the accuracy of both of them. With Simultaneous Diversity & Accuracy, those difficult samples are not given up so as to further improve the generalization performance of the ensemble. The inspiration of devising Diversity-Focused-Two stems from the cognition that ensemble diversity attaches more importance to the difference among the classifiers in an ensemble. Finally, the proposal of Accuracy-Reinforcement reinforces the concern about ensemble accuracy. Extensive experiments verified the effectiveness and efficiency of the proposed three pruning measures. Through the investigation of this work, it is found that by considering diversity and accuracy simultaneously for ensemble pruning, well-performed selective ensemble with superior generalization capability can be acquired, which is the scientific value of this paper.

A systematic comparison of different object-based classification techniques using high spatial resolution imagery in agricultural environments

Geographic Object-Based Image Analysis (GEOBIA) is becoming more prevalent in remote sensing classification, especially for high-resolution imagery. Many supervised classification approaches are applied to objects rather than pixels, and several studies have been conducted to evaluate the performance of such supervised classification techniques in GEOBIA. However, these studies did not systematically investigate all relevant factors affecting the classification (segmentation scale, training set size, feature selection and mixed objects). In this study, statistical methods and visual inspection were used to compare these factors systematically in two agricultural case studies in China. The results indicate that Random Forest (RF) and Support Vector Machines (SVM) are highly suitable for GEOBIA classifications in agricultural areas and confirm the expected general tendency, namely that the overall accuracies decline with increasing segmentation scale. All other investigated methods except for RF and SVM are more prone to obtain a lower accuracy due to the broken objects at fine scales. In contrast to some previous studies, the RF classifiers yielded the best results and the k-nearest neighbor classifier were the worst results, in most cases. Likewise, the RF and Decision Tree classifiers are the most robust with or without feature selection. The results of training sample analyses indicated that the RF and adaboost. M1 possess a superior generalization capability, except when dealing with small training sample sizes. Furthermore, the classification accuracies were directly related to the homogeneity/heterogeneity of the segmented objects for all classifiers. Finally, it was suggested that RF should be considered in most cases for agricultural mapping.

An Improved Continuous-Action Extended Classifier Systems for Function Approximation

Due to their structural simplicity and superior generalization capability, Extended Classifier Systems (XCSs) are gaining popularity within the Artificial Intelligence community. In this study an improved XCS with continuous actions is introduced for function approximation purposes. The proposed XCSF uses “prediction zones,” rather than distinct “prediction values,” to enable multi-member match sets that would allow multiple rules to be evaluated per training step. It is shown that this would accelerate the training procedure and reduce the computational cost associated with the training phase. The improved XCSF is also shown to produce more accurate rules than the classical classifier system when it comes to approximating complex nonlinear functions.

Modeling Spectral Envelopes Using Restricted Boltzmann Machines and Deep Belief Networks for Statistical Parametric Speech Synthesis

This paper presents a new spectral modeling method for statistical parametric speech synthesis. In the conventional methods, high-level spectral parameters, such as mel-cepstra or line spectral pairs, are adopted as the features for hidden Markov model (HMM)-based parametric speech synthesis. Our proposed method described in this paper improves the conventional method in two ways. First, distributions of low-level, un-transformed spectral envelopes (extracted by the STRAIGHT vocoder) are used as the parameters for synthesis. Second, instead of using single Gaussian distribution, we adopt the graphical models with multiple hidden variables, including restricted Boltzmann machines (RBM) and deep belief networks (DBN), to represent the distribution of the low-level spectral envelopes at each HMM state. At the synthesis time, the spectral envelopes are predicted from the RBM-HMMs or the DBN-HMMs of the input sentence following the maximum output probability parameter generation criterion with the constraints of the dynamic features. A Gaussian approximation is applied to the marginal distribution of the visible stochastic variables in the RBM or DBN at each HMM state in order to achieve a closed-form solution to the parameter generation problem. Our experimental results show that both RBM-HMM and DBN-HMM are able to generate spectral envelope parameter sequences better than the conventional Gaussian-HMM with superior generalization capabilities and that DBN-HMM and RBM-HMM perform similarly due possibly to the use of Gaussian approximation. As a result, our proposed method can significantly alleviate the over-smoothing effect and improve the naturalness of the conventional HMM-based speech synthesis system using mel-cepstra.

Comparison of support vector machine, neural network, and CART algorithms for the land-cover classification using limited training data points

Support vector machine (SVM) was applied for land-cover characterization using MODIS time-series data. Classification performance was examined with respect to training sample size, sample variability, and landscape homogeneity (purity). The results were compared to two conventional nonparametric image classification algorithms: multilayer perceptron neural networks (NN) and classification and regression trees (CART). For 2001 MODIS time-series data, SVM generated overall accuracies ranging from 77% to 80% for training sample sizes from 20 to 800 pixels per class, compared to 67–76% and 62–73% for NN and CART, respectively. These results indicated that SVM’s had superior generalization capability, particularly with respect to small training sample sizes. There was also less variability of SVM performance when classification trials were repeated using different training sets. Additionally, classification accuracies were directly related to sample homogeneity/heterogeneity. The overall accuracies for the SVM algorithm were 91% (Kappa=0.77) and 64% (Kappa=0.34) for homogeneous and heterogeneous pixels, respectively. The inclusion of heterogeneous pixels in the training sample did not increase overall accuracies. Also, the SVM performance was examined for the classification of multiple year MODIS time-series data at annual intervals. Finally, using only the SVM output values, a method was developed to directly classify pixel purity. Approximately 65% of pixels within the Albemarle–Pamlico Basin study area were labeled as “functionally homogeneous” with an overall classification accuracy of 91% (Kappa=0.79). The results indicated a high potential for regional scale operational land-cover characterization applications.

Linear Programming SVM-ARMA$_{\rm 2K}$ With Application in Engine System Identification

As an emerging non-parametric modeling technique, the methodology of support vector regression blazed a new trail in identifying complex nonlinear systems with superior generalization capability and sparsity. Nevertheless, the conventional quadratic programming support vector regression can easily lead to representation redundancy and expensive computational cost. In this paper, by using the <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">l</i> <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> norm minimization and taking account of the different characteristics of autoregression (AR) and the moving average (MA), an innovative nonlinear dynamical system identification approach, linear programming SVM-ARMA <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2K</sub> , is developed to enhance flexibility and secure model sparsity in identifying nonlinear dynamical systems. To demonstrate the potential and practicality of the proposed approach, the proposed strategy is applied to identify a representative dynamical engine model.

Evolutionary artificial neural networks by multi-dimensional particle swarm optimization

In this paper, we propose a novel technique for the automatic design of Artificial Neural Networks (ANNs) by evolving to the optimal network configuration(s) within an architecture space. It is entirely based on a multi-dimensional Particle Swarm Optimization (MD PSO) technique, which re-forms the native structure of swarm particles in such a way that they can make inter-dimensional passes with a dedicated dimensional PSO process. Therefore, in a multidimensional search space where the optimum dimension is unknown, swarm particles can seek both positional and dimensional optima. This eventually removes the necessity of setting a fixed dimension a priori, which is a common drawback for the family of swarm optimizers. With the proper encoding of the network configurations and parameters into particles, MD PSO can then seek the positional optimum in the error space and the dimensional optimum in the architecture space. The optimum dimension converged at the end of a MD PSO process corresponds to a unique ANN configuration where the network parameters (connections, weights and biases) can then be resolved from the positional optimum reached on that dimension. In addition to this, the proposed technique generates a ranked list of network configurations, from the best to the worst. This is indeed a crucial piece of information, indicating what potential configurations can be alternatives to the best one, and which configurations should not be used at all for a particular problem. In this study, the architecture space is defined over feed-forward, fully-connected ANNs so as to use the conventional techniques such as back-propagation and some other evolutionary methods in this field. The proposed technique is applied over the most challenging synthetic problems to test its optimality on evolving networks and over the benchmark problems to test its generalization capability as well as to make comparative evaluations with the several competing techniques. The experimental results show that the MD PSO evolves to optimum or near-optimum networks in general and has a superior generalization capability. Furthermore, the MD PSO naturally favors a low-dimension solution when it exhibits a competitive performance with a high dimension counterpart and such a native tendency eventually yields the evolution process to the compact network configurations in the architecture space rather than the complex ones, as long as the optimality prevails.

Albemarle starts new Texas catalyst plant

Support vector machine (SVM) was applied for land-cover characterization using MODIS time-series data. Classification performance was examined with respect to training sample size, sample variability, and landscape homogeneity (purity). The results were compared to two conventional nonparametric image classification algorithms: multilayer perceptron neural networks (NN) and classification and regression trees (CART). For 2001 MODIS time-series data, SVM generated overall accuracies ranging from 77% to 80% for training sample sizes from 20 to 800 pixels per class, compared to 67–76% and 62–73% for NN and CART, respectively. These results indicated that SVM’s had superior generalization capability, particularly with respect to small training sample sizes. There was also less variability of SVM performance when classification trials were repeated using different training sets. Additionally, classification accuracies were directly related to sample homogeneity/heterogeneity. The overall accuracies for the SVM algorithm were 91% (Kappa = 0.77) and 64% (Kappa = 0.34) for homogeneous and heterogeneous pixels, respectively. The inclusion of heterogeneous pixels in the training sample did not increase overall accuracies. Also, the SVM performance was examined for the classification of multiple year MODIS time-series data at annual intervals. Finally, using only the SVM output values, a method was developed to directly classify pixel purity. Approximately 65% of pixels within the Albemarle–Pamlico Basin study area were labeled as “functionally homogeneous” with an overall classification accuracy of 91% (Kappa = 0.79). The results indicated a high potential for regional scale operational land-cover characterization applications.

Granular support vector machines with association rules mining for protein homology prediction

Protein homology prediction between protein sequences is one of critical problems in computational biology. Such a complex classification problem is common in medical or biological information processing applications. How to build a model with superior generalization capability from training samples is an essential issue for mining knowledge to accurately predict/classify unseen new samples and to effectively support human experts to make correct decisions. A new learning model called granular support vector machines (GSVM) is proposed based on our previous work. GSVM systematically and formally combines the principles from statistical learning theory and granular computing theory and thus provides an interesting new mechanism to address complex classification problems. It works by building a sequence of information granules and then building support vector machines (SVM) in some of these information granules on demand. A good granulation method to find suitable granules is crucial for modeling a GSVM with good performance. In this paper, we also propose an association rules-based granulation method. For the granules induced by association rules with high enough confidence and significant support, we leave them as they are because of their high "purity" and significant effect on simplifying the classification task. For every other granule, a SVM is modeled to discriminate the corresponding data. In this way, a complex classification problem is divided into multiple smaller problems so that the learning task is simplified. The proposed algorithm, here named GSVM-AR, is compared with SVM by KDDCUP04 protein homology prediction data. The experimental results show that finding the splitting hyperplane is not a trivial task (we should be careful to select the association rules to avoid overfitting) and GSVM-AR does show significant improvement compared to building one single SVM in the whole feature space. Another advantage is that the utility of GSVM-AR is very good because it is easy to be implemented. More importantly and more interestingly, GSVM provides a new mechanism to address complex classification problems.

A GENETIC NEURAL FUZZY SYSTEM AND ITS APPLICATION IN QUALITY PREDICTION IN THE INJECTION PROCESS

A genetic neural fuzzy system (GNFS) is presented and introduced to quality prediction in the injection process. A hybrid-learning algorithm is proposed, which is divided into two stages to train GNFS. During the first learning stage, the genetic algorithm is used to optimize the structure of GNFS and the membership function of each fuzzy term because of its capability of parallel and global search. On the basis of the first optimized training stages, the back-propagation algorithm (BP algorithm) is adopted to update the parameters of the GNFS to improve its predicting precision and reduce the computation time. The process of constructing a quality prediction model for an injection process based on GNFS is described in detail. The predicted weight of the molded part from the model based on GNFS demonstrates that the proposed GNFS has superior performance and good generalization capability in quality prediction in the injection process.

A modified PNN algorithm with optimal PD modeling using the orthogonal least squares method

In this paper a modified algorithm is suggested for developing polynomial neural network (PNN) models. Optimal partial description (PD) modeling is introduced at each layer of the PNN expansion, a task accomplished using the orthogonal least squares (OLS) method. Based on the initial PD models determined by the polynomial order and the number of PD inputs, OLS selects the most significant regressor terms reducing the output error variance. The method produces PNN models exhibiting a high level of accuracy and superior generalization capabilities. Additionally, parsimonious models are obtained comprising a considerably smaller number of parameters compared to the ones generated by means of the conventional PNN algorithm. Three benchmark examples are elaborated, including modeling of the gas furnace process as well as the iris and wine classification problems. Extensive simulation results and comparison with other methods in the literature, demonstrate the effectiveness of the suggested modeling approach.

Runge-Kutta neural network for identification of dynamical systems in high accuracy

This paper proposes Runge-Kutta neural networks (RKNNs) for identification of unknown dynamical systems described by ordinary differential equations (i.e., ordinary differential equation or ODE systems) with high accuracy. These networks are constructed according to the Runge-Kutta approximation method. The main attraction of the RKNNs is that they precisely estimate the changing rates of system states (i.e., the right-hand side of the ODE x =f(x)) directly in their subnetworks based on the space-domain interpolation within one sampling interval such that they can do long-term prediction of system state trajectories. We show theoretically the superior generalization and long-term prediction capability of the RKNNs over the normal neural networks. Two types of learning algorithms are investigated for the RKNNs, gradient-and nonlinear recursive least-squares-based algorithms. Convergence analysis of the learning algorithms is done theoretically. Computer simulations demonstrate the proved properties of the RKNNs.

Superior Generalization Capability of Hardware-Learing Algorithm Developed for Self-Learning Neuron-MOS Neural Networks

We have investigated the learning performance of the hardware backpropagation (HBP) algorithm, a hardware-oriented learning algorithm developed for the self-learning architecture of neural networks constructed using neuron MOS (metal-oxide-semiconductor) transistors. The solution to finding a mirror symmetry axis in a 4×4 binary pixel array was tested by computer simulation based on the HBP algorithm. Despite the inherent restrictions imposed on the hardware-learning algorithm, HBP exhibits equivalent learning performance to that of the original backpropagation (BP) algorithm when all the pertinent parameters are optimized. Very importantly, we have found that HBP has a superior generalization capability over BP; namely, HBP exhibits higher performance in solving problems that the network has not yet learnt.

AN ADAPTIVE MODULAR NEURAL NETWORK WITH APPLICATION TO UNCONSTRAINED CHARACTER RECOGNITION

The topology and the capacity of a traditional multilayer neural system, as measured by the number of connections in the network, has surprisingly little impact on its generalization ability. This paper presents a new adaptive modular network that offers superior generalization capability. The new network provides significant fault tolerance, quick adaption to novel inputs, and high recognition accuracy. We demonstrate this paradigm on recognition of unconstrained handwritten characters.