High Generalization Accuracy Research Articles

A novel semi-supervised robust learning framework for dynamic generative latent variable models and its application to industrial virtual metrology

Among a variety of virtual metrology models, dynamic generative latent variable models (DGLVMs) have proven to be an effective tool, owing to outstanding advantages in dealing with complicated data characteristics including temporal correlations, variable colinearities, high dimensionality, and process and measurement uncertainties. Presently, the vast majority of DGLVMs are developed upon the assumption that data are Gaussian-distributed, which is however, non-robust against outlying data and leads the DGLVMs to compromised performance, as the Gaussian distribution is susceptible to outliers that would skew the estimated parameters of DGLVMs. In view of this, this paper proposes a semi-supervised robust learning (SsRL) framework for the DGLVMs by designing an ad-hoc fully robust model structure that handles the outlying data by means of the heavy-tail properties of Student’s t distribution. In addition, the SsRL-DGLVM framework is initiated as a concrete model using probabilistic slow feature analysis (denoted RoSsPSFA) for which a rigorous training algorithm without resorting to any approximations is developed. The performance of the RoSsPSFA is thoroughly evaluated by both numerical and real-world industrial cases, showing the RoSsPSFA’s better immunity against outlying data and higher generalization accuracy.

Extracting fetal heart signals from Doppler using semi-supervised convolutional neural networks.

Cardiotocography (CTG) measurements are critical for assessing fetal wellbeing during monitoring, and accurate assessment requires well-traceable CTG signals. The current FHR calculation algorithm, based on autocorrelation to Doppler ultrasound (DUS) signals, often results in periods of loss owing to its inability to differentiate signals. We hypothesized that classifying DUS signals by type could be a solution and proposed that an artificial intelligence (AI)-based approach could be used for classification. However, limited studies have incorporated the use of AI for DUS signals because of the limited data availability. Therefore, this study focused on evaluating the effectiveness of semi-supervised learning in enhancing classification accuracy, even in limited datasets, for DUS signals. Data comprising fetal heartbeat, artifacts, and two other categories were created from non-stress tests and labor DUS signals. With labeled and unlabeled data totaling 9,600 and 48,000 data points, respectively, the semi-supervised learning model consistently outperformed the supervised learning model, achieving an average classification accuracy of 80.9%. The preliminary findings indicate that applying semi-supervised learning to the development of AI models using DUS signals can achieve high generalization accuracy and reduce the effort. This approach may enhance the quality of fetal monitoring.

Prediction approach of larch wood density from visible-near-infrared spectroscopy based on parameter calibrating and transfer learning.

Wood density, as a key indicator to measure wood properties, is of weighty significance in enhancing wood utilization and modifying wood properties in sustainable forest management. Visible–near-infrared (Vis-NIR) spectroscopy provides a feasible and efficient solution for obtaining wood density by the advantages of its efficiency and non-destructiveness. However, the spectral responses are different in wood products with different moisture content conditions, and changes in external factors may cause the regression model to fail. Although some calibration transfer methods and convolutional neural network (CNN)-based deep transfer learning methods have been proposed, the generalization ability and prediction accuracy of the models still need to be improved. For the prediction problem of Vis-NIR wood density in different moisture contents, a deep transfer learning hybrid method with automatic calibration capability (Resnet1D-SVR-TrAdaBoost.R2) was proposed in this study. The disadvantage of overfitting was avoided when CNN processes small sample data, which considered the complex exterior factors in actual production to enhance feature extraction and migration between samples. Density prediction of the method was performed on a larch dataset with different moisture content conditions, and the hybrid method was found to achieve the best prediction results under the calibration samples with different target domain calibration samples and moisture contents, and the performance of models was better than that of the traditional calibration transfer and migration learning methods. In particular, the hybrid model has achieved an improvement of about 0.1 in both R 2 and root mean square error (RMSE) values compared to the support vector regression model transferred by piecewise direct standardization method (SVR+PDS), which has the best performance among traditional calibration methods. To further ascertain the generalizability of the hybrid model, the model was validated with samples collected from mixed moisture contents as the target domain. Various experiments demonstrated that the Resnet1D-SVR-TrAdaBoost.R2 model could predict larch wood density with a high generalization ability and accuracy effectively but was computation consuming. It showed the potential to be extended to predict other metrics of wood.

MetaMed: Few-shot medical image classification using gradient-based meta-learning

The occurrence of long-tailed distributions and unavailability of high-quality annotated images is a common phenomenon in medical datasets. The use of conventional Deep Learning techniques to obtain an unbiased model with high generalization accuracy for such datasets is a challenging task. Thus, we formulated a few-shot learning problem and presented a meta-learning-based “MetaMed” approach. The model presented here can adapt to rare disease classes with the availability of few images, and less compute. MetaMed is validated on three publicly accessible medical datasets – Pap smear, BreakHis, and ISIC 2018. We used advanced image augmentation techniques like CutOut, MixUp, and CutMix to overcome the problem of over-fitting. Our approach has shown promising results on all the three datasets with an accuracy of more than 70%. Inclusion of advanced augmentation techniques regularizes the model and increases the generalization capability by 2–5%. Comparative analysis of MetaMed against transfer learning demonstrated that MetaMed classifies images with a higher confidence score and on average outperforms transfer learning for 3, 5, and 10-shot tasks for both 2-way and 3-way classification.

Relationship between manifold smoothness and adversarial vulnerability in deep learning with local errors* *Project supported by the National Key R&D Program of China (Grant No. 2019YFA0706302), the start-up budget 74130-18831109 of the 100-talent-program of Sun Yat-sen University, and the National Natural Science Foundation of China (Grant No. 11805284).

Artificial neural networks can achieve impressive performances, and even outperform humans in some specific tasks. Nevertheless, unlike biological brains, the artificial neural networks suffer from tiny perturbations in sensory input, under various kinds of adversarial attacks. It is therefore necessary to study the origin of the adversarial vulnerability. Here, we establish a fundamental relationship between geometry of hidden representations (manifold perspective) and the generalization capability of the deep networks. For this purpose, we choose a deep neural network trained by local errors, and then analyze emergent properties of the trained networks through the manifold dimensionality, manifold smoothness, and the generalization capability. To explore effects of adversarial examples, we consider independent Gaussian noise attacks and fast-gradient-sign-method (FGSM) attacks. Our study reveals that a high generalization accuracy requires a relatively fast power-law decay of the eigen-spectrum of hidden representations. Under Gaussian attacks, the relationship between generalization accuracy and power-law exponent is monotonic, while a non-monotonic behavior is observed for FGSM attacks. Our empirical study provides a route towards a final mechanistic interpretation of adversarial vulnerability under adversarial attacks.

A Novel Multiway Splits Decision Tree for Multiple Types of Data

Classical decision trees such as C4.5 and CART partition the feature space using axis-parallel splits. Oblique decision trees use the oblique splits based on linear combinations of features to potentially simplify the boundary structure. Although oblique decision trees have higher generalization accuracy, most oblique split methods are not directly conducive to the categorical data and are computationally expensive. In this paper, we propose a multiway splits decision tree (MSDT) algorithm, which adopts feature weighting and clustering. This method can combine multiple numerical features, multiple categorical features, or multiple mixed features. Experimental results show that MSDT has excellent performance for multiple types of data.

Deep Vibro-Tactile Perception for Simultaneous Texture Identification, Slip Detection, and Speed Estimation.

Autonomous dexterous manipulation relies on the ability to recognize an object and detect its slippage. Dynamic tactile signals are important for object recognition and slip detection. An object can be identified based on the acquired signals generated at contact points during tactile interaction. The use of vibrotactile sensors can increase the accuracy of texture recognition and preempt the slippage of a grasped object. In this work, we present a Deep Learning (DL) based method for the simultaneous texture recognition and slip detection. The method detects non-slip and slip events, the velocity, and discriminate textures—all within 17 ms. We evaluate the method for three objects grasped using an industrial gripper with accelerometers installed on its fingertips. A comparative analysis of convolutional neural networks (CNNs), feed-forward neural networks, and long short-term memory networks confirmed that deep CNNs have a higher generalization accuracy. We also evaluated the performance of the highest accuracy method for different signal bandwidths, which showed that a bandwidth of 125 Hz is enough to classify textures with 80% accuracy.

A Robust and Accurate Deep-learning-based Method for the Segmentation of Subcortical Brain: Cross-dataset Evaluation of Generalization Performance.

Purpose:To analyze subcortical brain volume more reliably, we propose a deep learning segmentation method of subcortical brain based on magnetic resonance imaging (MRI) having high generalization performance, accuracy, and robustness.Methods:First, local images of three-dimensional (3D) bounding boxes were extracted for seven subcortical structures (thalamus, putamen, caudate, pallidum, hippocampus, amygdala, and accumbens) from a whole brain MR image as inputs to the neural network. Second, dilated convolution layers, which input information of variable scope, were introduced to the blocks that make up the neural network. These blocks were connected in parallel to simultaneously process global and local information obtained by the dilated convolution layers. To evaluate generalization performance, different datasets were used for training and testing sessions (cross-dataset evaluation) because subcortical brain segmentation in clinical analysis is assumed to be applied to unknown datasets.Results:The proposed method showed better generalization performance that can obtain stable accuracy for all structures, whereas the state-of-the-art deep learning method obtained extremely low accuracy for some structures. The proposed method performed segmentation for all samples without failing with significantly higher accuracy (P < 0.005) than conventional methods such as 3D U-Net, FreeSurfer, and Functional Magnetic Resonance Imaging of the Brain’s (FMRIB’s) Integrated Registration and Segmentation Tool in the FMRIB Software Library (FSL-FIRST). Moreover, when applying this proposed method to larger datasets, segmentation was robustly performed for all samples without producing segmentation results on the areas that were apparently different from anatomically relevant areas. On the other hand, FSL-FIRST produced segmentation results on the area that were apparently and largely different from the anatomically relevant area for about one-third to one-fourth of the datasets.Conclusion:The cross-dataset evaluation showed that the proposed method is superior to existing methods in terms of generalization performance, accuracy, and robustness.

A Novel Identification Method for Apple (Malus domestica Borkh.) Cultivars Based on a Deep Convolutional Neural Network with Leaf Image Input

The innovation of germplasm resources and the continuous breeding of new varieties of apples (Malus domestica Borkh.) have yielded more than 8000 apple cultivars. The ability to identify apple cultivars with ease and accuracy can solve problems in apple breeding related to property rights protection to promote the healthy development of the global apple industry. However, the existing methods are inconsistent and time-consuming. This paper proposes an efficient and convenient method for the classification of apple cultivars using a deep convolutional neural network with leaf image input, which is the delicate symmetry of a human brain learning. The model was constructed using the TensorFlow framework and trained on a dataset of 12,435 leaf images for the identification of 14 apple cultivars. The proposed method achieved an overall accuracy of 0.9711 and could successfully avoid the over-fitting problem. Tests on an unknown independent testing set resulted in a mean accuracy, mean error, and variance of μ a c c = 0.9685 , μ ε = 0.0315 , and σ 2 = 1.89025 E − 4 , respectively, indicating that the generalization accuracy and stability of the model were very good. Finally, the classification performance for each cultivar was tested. The results show that model had an accuracy of 1.0000 for Ace, Hongrouyouxi, Jazz, and Honey Crisp cultivars, and only one leaf was incorrectly identified for 2001, Ada Red, Jonagold, and Gold Spur cultivars, with accuracies of 0.9787, 0.9800, 0.9773, and 0.9737, respectively. Jingning1 and Pinova cultivars were classified with the lowest accuracies, with 0.8780 and 0.8864, respectively. The results also show that the genetic relationship between cultivars Shoufu 3 and Yanfu 3 is very high, which is mainly because they were both selected from a red mutation of Fuji and bred in Yantai City, Shandong Province, China. Generally, this study indicates that the proposed deep learning model is a novel and improved solution for apple cultivar identification, with high generalization accuracy, stable convergence, and high specificity.

Multi-Factor Operating Condition Recognition Using 1D Convolutional Long Short-Term Network.

For a diesel engine, operating conditions have extreme importance in fault detection and diagnosis. Limited to various special circumstances, the multi-factor operating conditions of a diesel engine are difficult to measure, and the demand of automatic condition recognition based on vibration signals is urgent. In this paper, multi-factor operating condition recognition using a one-dimensional (1D) convolutional long short-term network (1D-CLSTM) is proposed. Firstly, a deep neural network framework is proposed based on a 1D convolutional neural network (CNN) and long short-Term network (LSTM). According to the characteristics of vibration signals of a diesel engine, batch normalization is introduced to regulate the input of each convolutional layer by fixing the mean value and variance. Subsequently, adaptive dropout is proposed to improve the model sparsity and prevent overfitting in model training. Moreover, the vibration signals measured under 12 operating conditions were used to verify the performance of the trained 1D-CLSTM classifier. Lastly, the vibration signals measured from another kind of diesel engine were applied to verify the generalizability of the proposed approach. Experimental results show that the proposed method is an effective approach for multi-factor operating condition recognition. In addition, the adaptive dropout can achieve better training performance than the constant dropout ratio. Compared with some state-of-the-art methods, the trained 1D-CLSTM classifier can predict new data with higher generalization accuracy.

A two ensemble system to handle concept drifting data streams: recurring dynamic weighted majority

We present an ensemble system, recurring dynamic weighted majority (RDWM) that maintains two ensembles of experts, so as to accurately handle drifting concepts mainly recurrent drifts. The primary online ensemble represents the present concepts and the secondary ensemble represents the old concepts since the beginning of learning. An effective pruning methodology helps to remove redundant and old classifiers, which may have otherwise caused interference in learning the new concepts. Experimental evaluation using datasets proves that RDWM achieves very high generalization accuracy, irrespective of the speed or severity of drift; or presence of noise in the dataset.

Extracting optimal actionable plans from additive tree models

Although amazing progress has been made in machine learning to achieve high generalization accuracy and efficiency, there is still very limited work on deriving meaningful decision-making actions from the resulting models. However, in many applications such as advertisement, recommendation systems, social networks, customer relationship management, and clinical prediction, the users need not only accurate prediction, but also suggestions on actions to achieve a desirable goal (e.g., high ads hit rates) or avert an undesirable predicted result (e.g., clinical deterioration). Existing works for extracting such actionability are few and limited to simple models such as a decision tree. The dilemma is that those models with high accuracy are often more complex and harder to extract actionability from. In this paper, we propose an effective method to extract actionable knowledge from additive tree models (ATMs), one of the most widely used and best off-the-shelf classifiers. We rigorously formulate the optimal actionable planning (OAP) problem for a given ATM, which is to extract an actionable plan for a given input so that it can achieve a desirable output while maximizing the net profit. Based on a state space graph formulation, we first propose an optimal heuristic search method which intends to find an optimal solution. Then, we also present a sub-optimal heuristic search with an admissible and consistent heuristic function which can remarkably improve the efficiency of the algorithm. Our experimental results demonstrate the effectiveness and efficiency of the proposed algorithms on several real datasets in the application domain of personal credit and banking.

Development of a Self-Regulating Evolving Spiking Neural Network for classification problem

This paper presents a new spiking neural network for pattern classification problems, referred to as the Self-Regulating Evolving Spiking Neural (SRESN) classifier, that regulates the learning process of the network. It uses a two layered spiking neural network and the input layer consists of receptive field neurons, which convert a real valued input to spikes using the population coding scheme without any delays. The output layer consists of leaky integrate-and-fire neurons. Since SRESN does not use any delays, the number of network parameters for SRESN is significantly lower than that used by other spiking neural networks, used in this study. During training, the learning algorithm for SRESN, automatically evolves neurons in the output layer based on the training data stream and the current knowledge stored in the network. Depending on the knowledge in the sample and the class specific knowledge stored in the network, it can choose to either add a neuron or update the network parameters or skip learning the sample resulting in self-regulation of the learning process. In case of neuron addition, the weights for the newly added neuron are initialized using a modified rank order scheme which facilitates SRESN for use in online/sequential as well as batch learning modes. The parameter update strategy in SRESN ensures that connections with non-zero postsynaptic potential at the time of the spike are alone updated which helps prevent over training. While evaluating the performance of SRESN, first a study is conducted to assess the impact of various parameters on its performance and establish guidelines to choose suitable values for these parameters. Next, the performance of SRESN, operating in batch mode, is compared with other spiking neural classifiers, including SpikeProp and MuSpiNN, for the UCI benchmark problems of Iris flower classification and Wisconsin breast cancer. Subsequently, the performance of SRESN in online and batch learning mode is compared with an evolving spiking neural classifier for five benchmark data sets from the UCI machine learning repository. Finally, SRESN is applied to solve the practical problem of Epilepsy detection. The performance comparison clearly indicates that SRESN provides a higher generalization accuracy using fewer network parameters.

Maximin Separation Probability Clustering

This paper proposes a new approach for discriminative clustering. The intuition is, for a good clustering, one should be able to learn a classifier from the clustering labels with high generalization accuracy. Thus we define a novel metric to evaluate the quality of a clustering labeling, named Minimum Separation Probability (MSP), which is a lower bound of the generalization accuracy of a classifier learnt from the clustering labeling. We take MSP as the objective to maximize and propose our approach Maximin Separation Probability Clustering (MSPC), which has several attractive properties, such as invariance to anisotropic feature scaling and intuitive probabilistic explanation for clustering quality. We present three efficient optimization strategies for MSPC, and analyze their interesting connections to existing clustering approaches, such as maximum margin clustering (MMC) and discriminative k-means. Empirical results on real world data sets verify that MSP is a robust and effective clustering quality measure. It is also shown that the proposed algorithms compare favorably to state-of-the-art clustering algorithms in both accuracy and efficiency.

A novel intrusion detection system based on feature generation with visualization strategy

In this paper, a four-angle-star based visualized feature generation approach, FASVFG, is proposed to evaluate the distance between samples in a 5-class classification problem. Based on the four angle star image, numerical features are generated for network visit data from KDDcup99, and an efficient intrusion detection system with less features is proposed. The FASVFG-based classifier achieves a high generalization accuracy of 94.3555% in validation experiment, and the average Mathews correlation coefficient reaches 0.8858.

Online ship roll motion prediction based on grey sequential extreme learning machine

For the online prediction of nonlinear systems with characteristics of time-varying dynamics and uncertainty, a sequential grey prediction approach is proposed based on the online sequential extreme learning machine (OS-ELM). The grey processing of time series alleviates the unfavorable effects of uncertainty in measurement data; the extremely fast learning speed and high generalization accuracy of OS-ELM enable online application of the sequential grey prediction approach. Ship's roll motion at sea is a complex nonlinear process with time-varying dynamics. Its dynamics also involves uncertainty caused by wind, random waves and rudder actions. In this paper, the proposed OS-ELM-based grey prediction approach is implemented for online ship roll prediction. The simulation of prediction is based on measurement data obtained from sea trials of the scientific research and training ship Yu Kun. Simulation results of ship roll prediction demonstrate the effectiveness and efficiency of the proposed grey neural prediction approach in dealing with time-varying nonlinear system with uncertainty.

Toward open sharing of task-based fMRI data: the OpenfMRI project

The large-scale sharing of task-based functional neuroimaging data has the potential to allow novel insights into the organization of mental function in the brain, but the field of neuroimaging has lagged behind other areas of bioscience in the development of data sharing resources. This paper describes the OpenFMRI project (accessible online at http://www.openfmri.org), which aims to provide the neuroimaging community with a resource to support open sharing of task-based fMRI studies. We describe the motivation behind the project, focusing particularly on how this project addresses some of the well-known challenges to sharing of task-based fMRI data. Results from a preliminary analysis of the current database are presented, which demonstrate the ability to classify between task contrasts with high generalization accuracy across subjects, and the ability to identify individual subjects from their activation maps with moderately high accuracy. Clustering analyses show that the similarity relations between statistical maps have a somewhat orderly relation to the mental functions engaged by the relevant tasks. These results highlight the potential of the project to support large-scale multivariate analyses of the relation between mental processes and brain function.

Network Anomaly Detection via Clustering and Custom Kernel in MSVM

Multiclass Support Vector Machines (MSVM) have been applied to build classifiers, which can help Network Intrusion detection. Beside their high generalization accuracy, the learning time of MSVM classifiers is still a concern when applied into Network intrusion detection systems. This paper speeds up the learning time of MSVM classifiers by reducing the number of support vectors. In this study, we proposed KMSVM method combines the K-means clustering technique with custom kernel in MSVM. Experiments performed on KDD99 dataset using KMSVM method, and the results show that the KMSVM method can speed up the learning time of classifiers by both reducing support vectors and improve the detection rate on testing dataset.

Novel network architecture and learning algorithm for the classification of mass abnormalities in digitized mammograms

The main objective of this paper is to present a novel learning algorithm for the classification of mass abnormalities in digitized mammograms. The proposed approach consists of new network architecture and a new learning algorithm. The original idea is based on the introduction of an additional neuron in the hidden layer for each output class. The additional neurons for benign and malignant classes help in improving memorization ability without destroying the generalization ability of the network. The training is conducted by combining minimal distance-based similarity/random weights and direct calculation of output weights. The proposed approach can memorize training patterns with 100% retrieval accuracy as well as achieve high generalization accuracy for patterns which it has never seen before. The grey-level and breast imaging reporting and data system-based features from digitized mammograms are extracted and used to train the network with the proposed architecture and learning algorithm. The best results achieved by using the proposed approach are 100% on training set and 94% on test set. The proposed approach produced very promising results. It has outperformed existing classification approaches in terms of classification accuracy, generalization and memorization abilities, number of iterations, and guaranteed training on a benchmark database.

A new intrusion detection system using support vector machines and hierarchical clustering

Whenever an intrusion occurs, the security and value of a computer system is compromised. Network-based attacks make it difficult for legitimate users to access various network services by purposely occupying or sabotaging network resources and services. This can be done by sending large amounts of network traffic, exploiting well-known faults in networking services, and by overloading network hosts. Intrusion Detection attempts to detect computer attacks by examining various data records observed in processes on the network and it is split into two groups, anomaly detection systems and misuse detection systems. Anomaly detection is an attempt to search for malicious behavior that deviates from established normal patterns. Misuse detection is used to identify intrusions that match known attack scenarios. Our interest here is in anomaly detection and our proposed method is a scalable solution for detecting network-based anomalies. We use Support Vector Machines (SVM) for classification. The SVM is one of the most successful classification algorithms in the data mining area, but its long training time limits its use. This paper presents a study for enhancing the training time of SVM, specifically when dealing with large data sets, using hierarchical clustering analysis. We use the Dynamically Growing Self-Organizing Tree (DGSOT) algorithm for clustering because it has proved to overcome the drawbacks of traditional hierarchical clustering algorithms (e.g., hierarchical agglomerative clustering). Clustering analysis helps find the boundary points, which are the most qualified data points to train SVM, between two classes. We present a new approach of combination of SVM and DGSOT, which starts with an initial training set and expands it gradually using the clustering structure produced by the DGSOT algorithm. We compare our approach with the Rocchio Bundling technique and random selection in terms of accuracy loss and training time gain using a single benchmark real data set. We show that our proposed variations contribute significantly in improving the training process of SVM with high generalization accuracy and outperform the Rocchio Bundling technique.