Localized Generalization Error Research Articles

A Robust Frequency-Domain-Based Graph Adaptive Network for Parkinson's Disease Detection From Gait Data

Parkinson's disease (PD) is a neurodegenerative disease with a high incidence rate. Effective early diagnosis of PD is critical to prevent further deterioration of a patient's condition, where gait abnormalities are important factors for doctors to diagnose PD. Deep learning (DL)-based methods for PD detection using gait information recorded by non-invasive sensors have emerged to assist doctors in accurate and efficient disease diagnosis. However, most existing DL-based PD detection models neglect information in the frequency domain and do not adaptively model the correlation of signals among sensors. Moreover, different people have different gait patterns. Therefore, the generalization capabilities of PD detection models on diversities of individuals' gaits are essential. This work proposes a novel robust frequency-domain-based graph adaptive network (RFdGAD) for PD detection from gait information (i.e., vertical ground reaction force signals recorded by foot sensors). Specifically, the RFdGAD first learns the frequency-domain features of signals from each foot sensor by a frequency representation learning block. Then, the RFdGAD utilizes a graph adaptive network block taking frequency-domain features as input to adaptively learn and exploit the interconnection between different sensor signals for accurate PD detection. Moreover, the RFdGAD is trained by minimizing the proposed Jensen-Shannon divergence-based localized generalization error to improve the generalization performance of RFdGAD on unseen subjects. Experimental results show that the RFdGAD outperforms existing DL-based models for PD detection on three widely used datasets in terms of three metrics, including accuracy, F1-score, and geometric mean.

Multi-view deep forecasting for hourly solar irradiance with error correction

Short-term solar irradiance forecasting is crucial in managing power network operations and solar photovoltaic applications. In this paper, a Multi-view Deep Forecasting method with Error Correction (MvDF_EC) for 1-hour ahead solar forecasting is proposed. MvDF_EC comprises of the Multi-view Deep Forecasting method (MvDF) and a robust Radial Basis Function Neural Network trained via minimizing the Localized Generalization Error for compensating the solar forecasting error of MvDF. MvDF consists of three deep neural networks which learn representations of input data from different views. The three views are 1) the hierarchical local temporal information extracted by the Temporal Convolutional Neural Network (TCN), 2) the key context sequential information captured by the Bi-directional Long Short-Term Memory Neural Network with Temporal Attention (BLSTMattn), and 3) long-term temporal dependencies between local temporal patterns filtered by the Convolutional Gated Recurrent Unit Neural Network (C_GRU). The solar forecasting performance of the proposed MvDF_EC is evaluated with the National Solar Radiation Database. Simulation results show that MvDF_EC yields the most accurate solar prediction compared with the benchmarks including the smart persistence and the state-of-the-art models. The lowest relative Root Mean Square Error values for Maraba and Labelle are 22.08% and 27.40%, respectively in 1-hour ahead solar forecasting.

Multi-Localized Sensitive Autoencoder-Attention-LSTM For Skeleton-based Action Recognition

One of key challenges of skeleton-based action recognition (SAR) tasks is the complex nature of human motion patterns. Variations such as performers and viewpoints may impose negative effects to the action recognition accuracy. In this work, we propose the Multi-Localized Sensitive Autoencoder-Attention-LSTM (Multi-LiSAAL) for SAR. The Localized Stochastic Sensitive Autoencoder (LiSSA) encodes both spatial and temporal information, and extracts meaningful features from different parts (four limbs and a trunk) from the skeleton. The LiSSA is trained by minimizing the localized generalization error to enhance the robustness of autoencoders via reducing its sensitivity with respect to small variations in inputs. We apply an attention mechanism to assign different weights to different skeleton parts and focus more on informative sections. Then, a backbone classifier network takes weighted features as inputs to differentiates actions. Experimental results on five public benchmarking datasets show that the Multi-LiSAAL outperforms state-of-the-art methods.

Maximizing minority accuracy for imbalanced pattern classification problems using cost-sensitive Localized Generalization Error Model

Traditional machine learning methods may not yield satisfactory generalization capability when samples in different classes are imbalanced. These methods tend to sacrifice the accuracy of the minority class to improve the overall accuracy without regarding the fact that misclassifications of minority samples usually costs more in many real world applications. Therefore, we propose a neural network training method via a minimization of the cost-sensitive localized generalization error-based objective function (c-LGEM) to achieve a better balance of error yielded by the minority and the majority classes. The c-LGEM emphasizes the minimization of the generalization error of the minority class in a cost-sensitive manner. Experimental results obtained on 16 UCI datasets show that neural networks trained by the c-LGEM yield better performance in comparison to the performance yielded by some existing methods.

Multi-View Neural Network Ensemble for Short and Mid-Term Load Forecasting

Accurate load forecasting is essential to the operation and planning of power systems and electricity markets. In this paper, an ensemble of radial basis function neural networks (RBFNNs) is proposed which is trained by minimizing the localized generalization error for short-term and mid-term load forecasting. Exogenous features and features extracted from load series (with long short-term memory networks and multi-resolution wavelet transform) in various timescales are used to train the ensemble of RBFNNs. Multiple RBFNNs are fused as an ensemble model with high generalization capability using a proposed weighted fusion method based on the localized generalization error model. Experimental results on three practical datasets show that compared with other forecasting methods, the proposed method reduces the mean absolute percentage error (MAPE), mean squared error (MSE), mean absolute error (MAE) by at least 0.12%, 8.46 (MW) <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> , 0.83 MW in mid-term load forecasting (i.e., to predict the daily peak load of next month), respectively, and reduces the MAPE, MSE by at least 0.19%, 2009.69 (MW) <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> and 0.30%, 3697.18 (MW) <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> in half-hour-ahead forecasting and day-ahead forecasting, respectively.

Multi-Occupancy Fall Detection Using Non-Invasive Thermal Vision Sensor

Falling is a common issue within the aging population. The immediate detection of a fall is key to guarantee early and immediate attention to avoid other potential immobility risks and reduction in recovery time. Video-based approaches for monitoring fall detection, although being highly accurate, are largely perceived as being intrusive if deployed within living environments. As an alternative, thermal vision-based methods can be deployed to offer a more acceptable level of privacy. To date, thermal vision-based fall detection methods have largely focused on single-occupancy scenarios, which are not fully representative of real living environments with multi-occupancy. This work proposes a non-invasive thermal vision-based approach of multi-occupancy fall detection (MoT-LoGNN) which discriminates between a fall or no-fall. The approach consists of four major components: i) a multi-occupancy decomposer, ii) a sensitivity-based sample selector, iii) the T-LoGNN for single-occupancy fall detection, and iv) a fine-tuning mechanism. The T-LoGNN consists of a robust neural network minimizing a Localized Generalization Error (L-GEM) and thermal image features extracted by a Convolutional Neural Network (CNN). Comparing to other methods, the MoT-LoGNN achieved the highest average accuracy of 98.39% within the context of a multi-occupancy fall detection experiment.

Evaluation of radial basis function neural network minimizing L-GEM for sensor-based activity recognition

Sensor-based activity recognition involves the automatic recognition of a user’s activity in a smart environment using computational methods. The use of wearable devices and video-based approaches have attracted considerable interest in ubiquitous computing. Nevertheless, these methods have limitations such as issues with privacy invasion, ethics, comfort and obtrusiveness. Environmental sensors are an increasingly promising consideration in the ubiquitous computing domain for long-term monitoring, as these devices are non-invasive to inhabitants, yet certain challenges remain with activity recognition in sensorised environments, for example, addressing the challenge of intraclass variation between activities and reasoning from low-level uncertain information. In an effort to address these challenges, this paper proposes and evaluates the performance of a Radial Basis Function Neural Network approach for activity recognition with environmental sensors. The model is trained using the Localized Generalization Error and focuses on the generalization ability by considering both the training error and stochastic sensitivity measure. This measures the network output fluctuation with respect to the minor perturbation of input, to address the tolerance of the low-level uncertain sensor data. This approach is compared with three benchmark Neural Network approaches, including a popular deep learning approach using an Autoencoder, and it is evaluated with a simulated dataset as well as a number of publicly available datasets. The proposed method has shown advantages over the other models for all four evaluated datasets. This paper provides insights into the importance of model generalization abilities and an initial analysis of the limitation of deep Neural Networks with respect to sensor-based activity recognition.

Training an extreme learning machine by localized generalization error model

Extreme learning machine (ELM) is a non-iterative algorithm for training single-hidden layer feed-forward networks, whose training speed is much faster than those of conventional neural networks. However, its objective is only to minimize the empirical risk, which may cause overfitting easily. To overcome this defect, this paper proposes a novel algorithm named LGE2LM based on the localized generalization error model which provides an upper bound of generalization error through adopting the stochastic sensitivity. LGE2LM aims to improve the generalization capability of ELM by using the regularization technique which makes a trade-off between the empirical risk and the stochastic sensitive measure. The essence of LGE2LM is a quadratic problem without iterative process. Similar to ELM, all the parameters of this new algorithm are obtained without tuning, which makes the efficiency of LGE2LM much higher than those of traditional neural networks. Several experiments conducted on both artificial and real-world datasets show that LGE2LM has much better generalization capability and stronger robustness in comparison with ELM.

A synthetic neighborhood generation based ensemble learning for the imbalanced data classification

Constructing effective classifiers from imbalanced datasets has emerged as one of the main challenges in the data mining community, due to its increased prevalence in various real-world domains. Ensemble solutions are quite often applied in this field for their ability to provide better classification ability than single classifier. However, most existing methods adopt data sampling to train the base classifier on balanced datasets, but not to directly enhance the diversity. Thus, the performance of the final classifier can be limited. This paper suggests a new ensemble learning that can address the class imbalance problem and promote diversity simultaneously. Inspired by the localized generalization error model, this paper generates some synthetic samples located within some local area of the training samples, and trains the base classifiers with the union of original training samples and synthetic neighborhoods samples. By controlling the number of generated samples, the base classifiers can be trained with balanced datasets. Meanwhile, as the generated samples can extend different parts of the original input space and can be quite different from the original training samples, the obtained base classifiers are guaranteed to be accurate and diverse. A thorough experimental study on 36 benchmark datasets was performed, and the experimental results demonstrated that our proposed method can deliver significant better performance than the state-of-the-art ensemble solutions for the imbalanced problems.

Multi-criteria decision making based architecture selection for single-hidden layer feedforward neural networks

Architecture selection is a fundamental problem in artificial neural networks, which could be treated as a decision making process that evaluates, ranks, and makes choices from a set of network structures. Traditional methods evaluate a network structure by designing a criterion based on a validation model or an error bound model. On one hand, the time complexity of a validation model is usually high; on the other hand, different validation models or error bound models may lead to different (even conflicting) results, which post challenges to the traditional single criterion-based architecture selection methods. In the area of decision making, many problems employed multiple criteria since the performance is better than using a single criterion. In this paper, we propose a multi-criteria decision making based architecture selection algorithm for single-hidden layer feedforward neural networks trained by extreme learning machine. Two criteria are incorporated into the selection process, i.e., training accuracy and the Q-value estimated by the localized generalization error model. The training accuracy reflects the capability of the model on correctly categorizing the known samples, and the Q-value estimated by localized generalization error model reflects the size of the neighbourhood of training samples in which the model can predict unseen samples with confidence. By achieving a trade-off between these two criteria, a new architecture selection algorithm is proposed. Experimental comparisons demonstrate the feasibility and effectiveness of the proposed method.

Evolving Ensemble Fuzzy Classifier

The concept of ensemble learning offers a promising avenue in learning from data streams under complex environments because it addresses the bias and variance dilemma better than its single model counterpart and features a reconfigurable structure, which is well suited to the given context. While various extensions of ensemble learning for mining non-stationary data streams can be found in the literature, most of them are crafted under a static base classifier and revisits preceding samples in the sliding window for a retraining step. This feature causes computationally prohibitive complexity and is not flexible enough to cope with rapidly changing environments. Their complexities are often demanding because it involves a large collection of offline classifiers due to the absence of structural complexities reduction mechanisms and lack of an online feature selection mechanism. A novel evolving ensemble classifier, namely Parsimonious Ensemble pENsemble, is proposed in this paper. pENsemble differs from existing architectures in the fact that it is built upon an evolving classifier from data streams, termed Parsimonious Classifier pClass. pENsemble is equipped by an ensemble pruning mechanism, which estimates a localized generalization error of a base classifier. A dynamic online feature selection scenario is integrated into the pENsemble. This method allows for dynamic selection and deselection of input features on the fly. pENsemble adopts a dynamic ensemble structure to output a final classification decision where it features a novel drift detection scenario to grow the ensemble structure. The efficacy of the pENsemble has been numerically demonstrated through rigorous numerical studies with dynamic and evolving data streams where it delivers the most encouraging performance in attaining a tradeoff between accuracy and complexity.

Creating diversity in ensembles using synthetic neighborhoods of training samples

Diversity among base classifiers is known to be a key driver for the construction of an effective ensemble classifier. Several methods have been proposed to construct diverse base classifiers using artificially generated training samples. However, in these methods, diversity is often obtained at the expense of the accuracy of base classifiers. Inspired by the localized generalization error model a new sample generation method is proposed in this study. When preparing different training sets for base classifiers, the proposed method generates samples located within limited neighborhoods of the corresponding training samples. The generated samples are different with the original training samples but they also expand different parts of the original training data. Learning these datasets can result in a set of base classifiers that are accurate in different regions of the input space as well as maintaining appropriate diversity. Experiments performed on 26 benchmark datasets showed that: (1) our proposed method significantly outperformed some state-of-the-art ensemble methods in term of the classification accuracy; (2) our proposed method was significantly more efficient that other sample generation based ensemble methods.

Improved sparse LSSVMS based on the localized generalization error model

The least squares support vector machine (LSSVM) is computationally efficient because it converts the quadratic programming problem in the training of SVM to a linear programming problem. The sparse LSSVM is proposed to promote the predictive speed and generalization capability. In this paper, two sparse LSSVM algorithms: the SMRLSSVM and the RQRLSSVM are proposed based on the Localized Generalization Error of the LSSVM. Experimental results show that the RQRLSSVM yields both better generalization capability and sparseness in comparison to other sparse LSSVM algorithms.

An improved early detection method of type-2 diabetes mellitus using multiple classifier system

The specific causes of complex diseases such as Type-2 Diabetes Mellitus (T2DM) have not yet been identified. Nevertheless, many medical science researchers believe that complex diseases are caused by a combination of genetic, environmental, and lifestyle factors. Detection of such diseases becomes an issue because it is not free from false presumptions and is accompanied by unpredictable effects. Given the greatly increased amount of data gathered in medical databases, data mining has been used widely in recent years to detect and improve the diagnosis of complex diseases. However, past research showed that no single classifier can be considered optimal for all problems. Therefore, in this paper, we focus on employing multiple classifier systems to improve the accuracy of detection for complex diseases, such as T2DM. We proposed a dynamic weighted voting scheme called multiple factors weighted combination for classifiers’ decision combination. This method considers not only the local and global accuracy but also the diversity among classifiers and localized generalization error of each classifier. We evaluated our method on two real T2DM data sets and other medical data sets. The favorable results indicated that our proposed method significantly outperforms individual classifiers and other fusion methods.

LG-Trader: Stock trading decision support based on feature selection by weighted localized generalization error model

Stock trading is an important financial activity of human society. Machine learning techniques are adopted to provide trading decision support by predicting the stock price or trading signals of the next day. Decisions are made by analyzing technical indices and fundamental analysis of companies. There are two major machine learning research problems for stock trading decision support: classifier architecture selection and feature selection. In this work, we propose the LG-Trader which will deal with these two problems simultaneously using a genetic algorithm minimizing a new Weighted Localized Generalization Error (wL-GEM). An issue being ignored in current machine learning based stock trading researches is the imbalance among buy, hold and sell decisions. Usually hold decision is the majority in comparison to both buy and sell decisions. So, the wL-GEM is proposed to balance classes by penalizing heavier for generalization error being made in minority classes. The feature selection based on wL-GEM helps to select most useful technical indices among choices for each stock. Experimental results demonstrate that the LG-Trader yields higher profits and rates of return in both stock and index trading.

Applying a new localized generalization error model to design neural networks trained with extreme learning machine

High accuracy and low overhead are two key features of a well-designed classifier for different classification scenarios. In this paper, we propose an improved classifier using a single-hidden layer feedforward neural network (SLFN) trained with extreme learning machine. The novel classifier first utilizes principal component analysis to reduce the feature dimension and then selects the optimal architecture of the SLFN based on a new localized generalization error model in the principal component space. Experimental and statistical results on the NSL-KDD data set demonstrate that the proposed classifier can achieve a significant performance improvement compared with previous classifiers.

HYPER-PARAMETER SELECTION FOR SPARSE LS-SVM VIA MINIMIZATION OF ITS LOCALIZED GENERALIZATION ERROR

Sparse LS-SVM yields better generalization capability and reduces prediction time in comparison to full dense LS-SVM. However, both methods require careful selection of hyper-parameters (HPS) to achieve high generalization capability. Leave-One-Out Cross Validation (LOO-CV) and k-fold Cross Validation (k-CV) are the two most widely used hyper-parameter selection methods for LS-SVMs. However, both fail to select good hyper-parameters for sparse LS-SVM. In this paper we propose a new hyper-parameter selection method, LGEM-HPS, for LS-SVM via minimization of the Localized Generalization Error (L-GEM). The L-GEM consists of two major components: empirical mean square error and sensitivity measure. A new sensitivity measure is derived for LS-SVM to enable the LGEM-HPS select hyper-parameters yielding LS-SVM with smaller training error and minimum sensitivity to minor changes in inputs. Experiments on eleven UCI data sets show the effectiveness of the proposed method for selecting hyper-parameters for sparse LS-SVM.

Dynamic fusion method using Localized Generalization Error Model

Multiple Classifier Systems (MCSs), which combine the outputs of a set of base classifiers, were proposed as a method to develop a more accurate classification system. One fundamental issue is how to combine the base classifiers. In this paper, a new dynamic fusion method named Localized Generalization Error Model Fusion Method (LFM) for MCSs is proposed. The Localized Generalization Error Model (L-GEM) has been used to estimate the local competence of base classifiers in MCSs. L-GEM provides a generalization error bound for unseen samples located within neighborhoods of testing samples. Base classifiers with lower generalization error bounds are assigned higher weights. In contrast to the current dynamic fusion methods, LFM estimates the local competence of base classifiers not only using the information of training error but also the sensitivity of classifier outputs. The additional effect of the sensitivity on the performance of model and the time complexity of the LFM are discussed and analyzed. Experimental results show that the MCSs using the LFM as a combination method outperform those using the other 21 dynamic fusion methods in terms of testing accuracy and time.

Architecture selection for networks trained with extreme learning machine using localized generalization error model

The initial localized generalization error model (LGEM) aims to find an upper bound of error between a target function and a radial basis function neural network (RBFNN) within a neighborhood of the training samples. The contribution of LGEM can be briefly described as that the generalization error is less than or equal to the summation of three terms: training error, stochastic sensitivity measure (SSM), and a constant. This paper extends the initial LGEM to a new LGEM model for single-hidden layer feed-forward neural networks (SLFNs) trained with extreme learning machine (ELM) which is a type of new training algorithms without iterations. The development of this extended LGEM can provide some useful guidelines for improving the generalization ability of SLFNs trained with ELM. An algorithm for architecture selection of the SLFNs is also proposed based on the extended LGEM. Experimental results on a number of benchmark data sets show that an approximately optimal architecture in terms of number of neurons of a SLFN can be found using our method. Furthermore, the experimental results on eleven UCI data sets show that the proposed method is effective and efficient.

Active Learning Based on New Localized Generalization Error Model for Training RBFNN

A new active learning based on a new localized generalization error model is proposed in the paper for training RBFNN. The samples with largest local generalization error are selected and labelled. The experimental results show that the proposed algorithm is effective, which can select the most informative samples and fewer samples are necessary.