Degrade Classification Performance Research Articles

Trainable pruned ternary quantization for medical signal classification models

The field of deep learning is renowned for its resource-intensive nature, hence improving its environmental impact is crucial. In this paper, we propose a novel model compression method to mitigate the energy demands of deep learning for a greener, and more sustainable AI landscape. Our approach relies on an asymmetric weakly-differentiable pruning function that leverages weight statistics to directly incorporate adaptable pruning into the quantization mechanism. This enables us to achieve higher compression rates globally while simultaneously reducing energy consumption and minimizing classification performance degradation. The efficacy of our approach was evaluated using three distinct models on three distinct datasets: cerebral emboli (HITS), epileptic seizure recognition (ESR), and MNIST. Our method demonstrated a superior balance between compression, energy consumption, and classification performance compared to other state-of-the-art extreme quantization methods, across all models and datasets. In fact, on the HITS dataset with a two-dimensional convolutional neural network, we achieved strong gains of 50.6%, 54.9%, 52.1% in compression rates (of the global model and the quantized layers only, respectively) and energy consumption, respectively, while improving the Matthews correlation coefficient by 2.5% compared to other approaches. The code is available at: https://github.com/yamilvindas/pTTQ.

Concept drift adaptation with continuous kernel learning

Concept drift poses significant challenges in the fields of machine learning and data mining. At present, many existing algorithms struggle to maintain low error rates or require excessive computational resources to achieve satisfactory classification results when addressing concept drift. To address these issues, a novel mathematical model is first introduced to mitigate the degradation of classification performance caused by concept drift. Then, based on the proposed objective function, a continuous kernel learning method is employed to adapt to potential changes in data distribution as new samples continuously arrive. Furthermore, we propose an ensemble learning approach leveraging the majority voting strategy to enhance classification performance in non-stationary environments. Finally, a theoretical analysis of the proposed algorithm is conducted. Experimental results demonstrate that the proposed algorithm not only achieves lower error rates and reduced memory consumption but also operates more efficiently than most of the state-of-the-art algorithms when processing different types of data streams.

Handling class imbalance and overlap with a Hesitation-based instance selection method

Class imbalance is a common problem in machine learning, particularly in classification tasks. When the distribution of instances across known classes is biased or skewed, this issue leads to poor predictive performance. This is especially true for the minority class, which is often of greater importance in problems. However, class imbalance is not the only factor that can decrease performance. Overlapping problems and borderline instances can also degrade classification performance. Conventional imbalanced learning methods often balance the distribution between classes, for example, by oversampling the minority class or under sampling the majority class. However, these methods may not adequately address the difficulties caused by overlapping and borderline instances. The present paper expands on the classification of imbalanced datasets by addressing the issue of how boundary instances should be sampled to handle class imbalance and control class overlap. The designed Hesitation degree-based instance weighting method can identify the impact of instances on classifier performance while reducing the skewness of the dataset and alleviating the possibility of class overlap. Additionally, by integrating a chaotic evolutionary algorithm with the designed classifier, the most important instances can be selected. Statistical results show that the proposed method outperforms state-of-the-art methods in terms of reduction rate, error rate, and G-mean.

Scaling Few-Shot Learning for the Open World

Few-shot learning (FSL) aims to enable learning models with the ability to automatically adapt to novel (unseen) domains in open-world scenarios. Nonetheless, there exists a significant disparity between the vast number of new concepts encountered in the open world and the restricted available scale of existing FSL works, which primarily focus on a limited number of novel classes. Such a gap hinders the practical applicability of FSL in realistic scenarios. To bridge this gap, we propose a new problem named Few-Shot Learning with Many Novel Classes (FSL-MNC) by substantially enlarging the number of novel classes, exceeding the count in the traditional FSL setup by over 500-fold. This new problem exhibits two major challenges, including the increased computation overhead during meta-training and the degraded classification performance by the large number of classes during meta-testing. To overcome these challenges, we propose a Simple Hierarchy Pipeline (SHA-Pipeline). Due to the inefficiency of traditional protocols of EML, we re-design a lightweight training strategy to reduce the overhead brought by much more novel classes. To capture discriminative semantics across numerous novel classes, we effectively reconstruct and leverage the class hierarchy information during meta-testing. Experiments show that the proposed SHA-Pipeline significantly outperforms not only the ProtoNet baseline but also the state-of-the-art alternatives across different numbers of novel classes.

Two-Stream spectral-spatial convolutional capsule network for Hyperspectral image classification

Recently, the capsule network and its enhanced version named convolutional capsule network (Conv-CapsNet) were applied to hyperspectral image (HSI) classification and delivered outstanding performance, attributing to the enriched capabilities of capsules for feature expression and complex spatial knowledge excavation. However, restricted by the locality of convolutions in primary feature extraction, the rich spectral-spatial diagnostic information found in HSIs were insufficiently utilized, which degrades classification performance to a certain degree. In addition, most capsule networks involve a complex structure with many parameters, which are haunted by a common bottleneck of limited available training samples. To tackle these challenges, this paper presents a novel two-stream spectral-spatial convolutional capsule network (TSCCN) for HSIs to boost accurate classification with small samples. Based on the advanced convolutional capsule concept, a two-stream Conv-CapsNet architecture is developed that reduces trainable parameters and mitigates overfitting when samples are insufficient. Specifically, a 1D Conv-CapsNet and a 2D Conv-CapsNet sharing similar structures are developed as accompanying streams to fully excavate favorable features from the continuous spectra and raw spectral-spatial cubes, with the separate features fused to generate joint deep features for classification. To overcome the locality of primary convolutions, an efficient structural information mining module (SIM) is designed for each stream, to learn complementary cross-channel and cross-kernel inter-dependencies for subsequent capsule learning and further improve classification performance. Experiments on three well-known HSI datasets demonstrate the superiority of the proposed algorithm over nine representative state-of-the-art deep classifiers and its efficacy under small sample conditions, where all the methods were implemented in the Tensor-Flow framework and repeated for 10 times.

Integration of Multikinds Imputation With Covariance Adaptation Based on Evidence Theory.

For incomplete data classification, missing attribute values are often estimated by imputation methods before building classifiers. The estimated attribute values are not actual attribute values. Thus, the distributions of data will be changed after imputing, and this phenomenon often results in degradation of classification performance. Here, we propose a new framework called integration of multikinds imputation with covariance adaptation (MICA) based on evidence theory (ET) to effectively deal with the classification problem with incomplete training data and complete test data. In MICA, we first employ different kinds of imputation methods to obtain multiple imputed training datasets. In general, the distributions of each imputed training dataset and test dataset will be different. A covariance adaptation module (CAM) is then developed to reduce the distribution difference of each imputed training dataset and test dataset. Then, multiple classifiers can be learned on the multiple imputed training datasets, and they are complementary to each other. For a test pattern, we can combine the multiple pieces of soft classification results yielded by these classifiers based on ET to obtain better classification performance. However, the reliabilities/weights of different imputed training datasets are usually different, so the soft classification results cannot be treated equally during fusion. We propose to use covariance difference across datasets and accuracy of imputed training data to estimate the weights. Finally, the soft classification results discounted by the estimated weights are combined by ET to make the final class decision. MICA was compared with a variety of related methods on several datasets, and the experimental results demonstrate that this new method can significantly improve the classification performance.

Estimating Per-Class Statistics for Label Noise Learning.

Real-world data may contain a considerable amount of noisily labeled examples, which usually mislead the training algorithm and result in degraded classification performance on test data. Therefore, Label Noise Learning (LNL) was proposed, of which one popular research trend focused on estimating the critical statistics (e.g., sample mean and sample covariance), to recover the clean data distribution. However, existing methods may suffer from the unreliable sample selection process or can hardly be applied to multi-class cases. Inspired by the centroid estimation theory, we propose Per-Class Statistic Estimation (PCSE), which establishes the quantitative relationship between the clean (first-order and second-order) statistics and the corresponding noisy statistics for every class. This relationship is further utilized to induce a generative classifier for model inference. Unlike existing methods, our approach does not require sample selection from the instance level. Moreover, our PCSE can serve as a general post-processing strategy applicable to various popular networks pre-trained on the noisy dataset for boosting their classification performance. Theoretically, we prove that the estimated statistics converge to their ground-truth values as the sample size increases, even if the label transition matrix is biased. Empirically, we conducted intensive experiments on various binary and multi-class datasets, and the results demonstrate that PCSE achieves more precise statistic estimation as well as higher classification accuracy when compared with state-of-the-art methods in LNL.

HSNF: Hybrid sampling with two-step noise filtering for imbalanced data classification

Imbalanced data classification has received much attention in machine learning, and many oversampling methods exist to solve this problem. However, these methods may suffer from insufficient noise filtering, overlap between synthetic and original samples, etc., resulting in degradation of classification performance. To this end, we propose a hybrid sampling with two-step noise filtering (HSNF) method in this paper, which consists of three modules. In the first module, HSNF denoises twice according to different noise discrimination mechanisms. Note that denoising mechanism is essentially based on the Euclidean distance between samples. Then in the second module, the minority class samples are divided into two categories, boundary samples and safe samples, respectively, and a portion of the boundary majority class samples are removed. In the third module, different oversampling methods are used to synthesize instances for boundary minority class samples and safe minority class samples. Experimental results on synthetic data and benchmark datasets demonstrate the effectiveness of HSNF in comparison with several popular methods. The code of HSNF will be released.

A membership-based resampling and cleaning algorithm for multi-class imbalanced overlapping data

Real-world datasets frequently have an imbalanced class distribution, which significantly degrades classification performance. However, some studies have suggested that the adverse effects of class imbalance occur only when datasets have other intrinsic characteristics (such as class overlap, noise, and data scarcity). Noise and class overlap have the greatest effect. To deal with other intrinsic characteristics that affect classification performance in a multi-class environment, such as class overlap, noise, and data scarcity, we propose a method that can directly handle multi-class overlapping data, called the membership-based multi-class resampling and cleaning (MC-MBRC) algorithm. The proposed method divides samples into safe, overlapping, and noisy areas based on their membership degrees, and then according to the influence of the samples in each area on classification performance, it performs various operations such as noise removal, interpolation oversampling, and energy-based cleaning of the overlapping region. An extensive comparison using various datasets shows that, compared with state-of-the-art methods, the proposed method makes significant statistical improvements in different classification performance metrics and is robust to data containing class overlap and label noise. Furthermore, for datasets that do not have complex intrinsic features, MC -MBRC will not significantly degrade classification performance.

Dimension selection for EEG classification in the SPD Riemannian space based on PSO

Several classification methods in the task of electroencephalogram (EEG) classification represent input features as symmetric positive definite (SPD) matrices. By translating the data into a Riemannian space, the classification methods have obtained superior performance; however, the high dimensionality of the data typically results in low computational efficiency and degraded classification performance. Therefore, this paper proposes a novel dimension selection method named dimension selection for SPD matrices on Riemannian manifold (DSSR). DSSR can eliminate redundant dimensions for SPD matrices in the Riemannian space based on the framework of particle swarm optimization (PSO). First, we treat one column of the SPD matrix and its corresponding row as a group, where upon the PSO is used to find optimal groups. Next, we use Riemannian classifiers to evaluate the goodness of the selected groups. Finally, the dimension selected SPD matrices are classified by the generalized learning Riemannian space quantization (GLRSQ). Experimental results on two motor imagery datasets show the superior performance of the proposed methods over EEG classification. Neuroscience experiments prove that the optimal dimensions selected by our proposed method are consistent with the findings of brain science research.

Multiclass Laplacian support vector machine with functional analysis of variance decomposition

In classification problems, acquiring a sufficient amount of labeled samples sometimes proves expensive and time-consuming, while unlabeled samples are relatively easier to obtain. The Laplacian Support Vector Machine (LapSVM) is one of the successful methods that learn better classification functions by incorporating unlabeled samples. However, since LapSVM was originally designed for binary classification, it can not be applied directly to multiclass classification problems commonly encountered in practice. Thus we derive an extension of LapSVM to multiclass classification problems using an appropriate multiclass formulation. Another problem with LapSVM is that irrelevant variables easily degrade classification performance. The irrelevant variables can increase the variance of predicted values and make the model difficult to interpret. Therefore, this paper also proposes the multiclass LapSVM with functional analysis of variance decomposition to identify relevant variables. Through comprehensive simulations and real-world datasets, we demonstrate the efficiency and improved classification performance of the proposed methods.

Crop classification based on multi-temporal PolSAR images with a single tensor network

Accurate and reliable discrimination of crop categories is a significant data source for agricultural monitoring and food security evaluation research. The convolutional neural network (CNN) model is one of the most popular classifiers for crop discrimination based on polarimetric synthetic aperture radar (PolSAR) data. However, it is better to avoid directly using large amounts of raw features that extracted from PolSAR data for CNN models because of the “dimension disaster” problem caused by multiple periods and various feature extraction schemes. Consequently, an extra feature compression model has to be incorporated to mitigate the “dimension disaster” problem. However, ill coupling of the two models may result in degraded classification performance, thus the combination of two models has to be further optimized. In this paper, we propose a novel single tensor affine transformation network (TATN) for crop classification using multi-temporal PolSAR data, where the input sample of the network is a higher order tensor formed by raw features, and the hidden layers of the network adopt the tensor affine transformation rather than convolution to extract discriminative features for classification. Since the tensor affine transformation preserves the structural information of the original input tensor samples, the TATN is expected to achieve a higher crop classification accuracy. Moreover, the TATN holds less amount of parameters than most of deep learning models, which enables to avoid the extra feature compression procedure. The experimental results validate the merits of our model.

Feature selection for domain adaptation using complexity measures and swarm intelligence

Particle Swarm Optimization is an optimization algorithm that mimics the behaviour of a flock of birds, setting multiple particles that explore the search space guided by a fitness function in order to find the best possible solution. We apply the Sticky Binary Particle Swarm Optimization algorithm to perform feature selection for domain adaptation, a specific type of transfer learning in which the source and the target domain have a common feature space, a common task, but different distributions. When applying Particle Swarm Optimization, classification error is usually employed in the fitness function to evaluate the goodness of subsets of features. In this paper, we aim to compare this approach with using complexity metrics instead, under the assumption that reducing the complexity of the problem will lead to results that are independent from the classifier used for testing while being less computationally demanding. Therefore, we carried out experiments to compare the performance of both approaches in terms of classification accuracy, speed and number of features selected. We found out that our proposal, although in some cases incurs in a slight degradation of classification performance, it is indeed faster and selects fewer features, making it a feasible trade-off.

Impact of Rubin Observatory Cadence Choices on Supernovae Photometric Classification

The Vera C. Rubin Observatory’s Legacy Survey of Space and Time (LSST) will discover an unprecedented number of supernovae (SNe), making spectroscopic classification for all the events infeasible. LSST will thus rely on photometric classification, whose accuracy depends on the not-yet-finalized LSST observing strategy. In this work, we analyze the impact of cadence choices on classification performance using simulated multiband light curves. First, we simulate SNe with an LSST baseline cadence, a nonrolling cadence, and a presto-color cadence, which observes each sky location three times per night instead of twice. Each simulated data set includes a spectroscopically confirmed training set, which we augment to be representative of the test set as part of the classification pipeline. Then we use the photometric transient classification library snmachine to build classifiers. We find that the active region of the rolling cadence used in the baseline observing strategy yields a 25% improvement in classification performance relative to the background region. This improvement in performance in the actively rolling region is also associated with an increase of up to a factor of 2.7 in the number of cosmologically useful Type Ia SNe relative to the background region. However, adding a third visit per night as implemented in presto-color degrades classification performance due to more irregularly sampled light curves. Overall, our results establish desiderata on the observing cadence related to classification of full SNe light curves, which in turn impacts photometric SNe cosmology with LSST.

Flexible model weighting for one-dependence estimators based on point-wise independence analysis

Recent studies have shown that Bayesian network classifiers (BNCs) are powerful tools for knowledge representation and classification, and averaged one-dependence estimators (AODE) is one of the most popular and effective BNCs since it can achieve the tradeoff between bias and variance due to its independence assumptions and ensemble learning strategy. However, unverified independence assumptions may result in biased estimates of probability distribution and then degradation in classification performance. In this paper, we prove theoretically the uncertainty of probability-theoretic independence and propose to measure the independence between attribute values implicated in specific instance. The estimates of conditional probability can be finely tuned based on point-wise independence analysis. Point-wise log likelihood function is then applied as weighting metric for committee members of AODE to improve the estimate of joint probability. Extensive experiments on 36 benchmark datasets show that, compared to other state-of-the-art classifiers, weighted one-dependence estimators using point-wise independence analysis can achieve competitive classification performance in terms of zero-one loss, RMSE, bias-variance decomposition and conditional log likelihood.

Multi-site cross-organ calibrated deep learning (MuSClD): Automated diagnosis of non-melanoma skin cancer.

Although deep learning (DL) has demonstrated impressive diagnostic performance for a variety of computational pathology tasks, this performance often markedly deteriorates on whole slide images (WSI) generated at external test sites. This phenomenon is due in part to domain shift, wherein differences in test-site pre-analytical variables (e.g., slide scanner, staining procedure) result in WSI with notably different visual presentations compared to training data. To ameliorate pre-analytic variances, approaches such as CycleGAN can be used to calibrate visual properties of images between sites, with the intent of improving DL classifier generalizability. In this work, we present a new approach termed Multi-Site Cross-Organ Calibration based Deep Learning (MuSClD) that employs WSIs of an off-target organ for calibration created at the same site as the on-target organ, based off the assumption that cross-organ slides are subjected to a common set of pre-analytical sources of variance. We demonstrate that by using an off-target organ from the test site to calibrate training data, the domain shift between training and testing data can be mitigated. Importantly, this strategy uniquely guards against potential data leakage introduced during calibration, wherein information only available in the testing data is imparted on the training data. We evaluate MuSClD in the context of the automated diagnosis of non-melanoma skin cancer (NMSC). Specifically, we evaluated MuSClD for identifying and distinguishing (a) basal cell carcinoma (BCC), (b) in-situ squamous cell carcinomas (SCC-In Situ), and (c) invasive squamous cell carcinomas (SCC-Invasive), using an Australian (training, n=85) and a Swiss (held-out testing, n=352) cohort. Our experiments reveal that MuSCID reduces the Wasserstein distances between sites in terms of color, contrast, and brightness metrics, without imparting noticeable artifacts to training data. The NMSC-subtyping performance is statistically improved as a result of MuSCID in terms of one-vs. rest AUC: BCC (0.92vs 0.87, p=0.01), SCC-In Situ (0.87vs 0.73, p=0.15) and SCC-Invasive (0.92vs 0.82, p=1e-5). Compared to baseline NMSC-subtyping with no calibration, the internal validation results of MuSClD (BCC (0.98), SCC-In Situ (0.92), and SCC-Invasive (0.97)) suggest that while domain shift indeed degrades classification performance, our on-target calibration using off-target tissue can safely compensate for pre-analytical variabilities, while improving the robustness of the model.

FEDA: A Nonlinear Subspace Projection Approach for Electronic Nose Data Classification

The electronic nose (e-nose) is susceptible to sensor drift and instrumental variation, which may result in distribution discrepancy in data collected, hence leading to classification performance degradation. It is necessary to apply domain adaptation to solve this problem. A nonlinear subspace projection approach named feature entropy domain adaption (FEDA) is proposed for domain adaptation for e-nose data classification. One important aspect of FEDA is that adversarial training is introduced to minimize the distribution discrepancy between source and target domains. No projection matrix and parameter fine-tuning are needed anymore in comparison with the popular linear subspace projection approaches. In addition, feature norm and conditional entropy are introduced into adversarial training in FEDA to reduce the decision boundary uncertainty and the overlap between classes, respectively. Experimental results show that the FEDA can deal with the distribution discrepancy of e-nose effectively, and can achieve satisfactory classification accuracy on various datasets. Source code can be found at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/threedteam/DA_FEDA</uri> .

A new attention-based 3D densely connected cross-stage-partial network for motor imagery classification in BCI

Objective. The challenge for motor imagery (MI) in brain-computer interface (BCI) systems is finding a reliable classification model that has high classification accuracy and excellent robustness. Currently, one of the main problems leading to degraded classification performance is the inaccuracy caused by nonstationarities and low signal-to-noise ratio in electroencephalogram (EEG) signals. Approach. This study proposes a novel attention-based 3D densely connected cross-stage-partial network (DCSPNet) model to achieve efficient EEG-based MI classification. This is an end-to-end classification model framework based on the convolutional neural network (CNN) architecture. In this framework, to fully utilize the complementary features in each dimension, the optimal features are extracted adaptively from the EEG signals through the spatial-spectral-temporal (SST) attention mechanism. The 3D DCSPNet is introduced to reduce the gradient loss by segmenting the extracted feature maps to strengthen the network learning capability. Additionally, the design of the densely connected structure increases the robustness of the network. Main results. The performance of the proposed method was evaluated using the BCI competition IV 2a and the high gamma dataset, achieving an average accuracy of 84.45% and 97.88%, respectively. Our method outperformed most state-of-the-art classification algorithms, demonstrating its effectiveness and strong generalization ability. Significance. The experimental results show that our method is promising for improving the performance of MI-BCI. As a general framework based on time-series classification, it can be applied to BCI-related fields.

A novel hybrid approach for feature selection enhancement: COVID-19 case study

Feature selection is a promising Artificial Intelligence technique for screening, analysing, predicting, and tracking current COVID-19 patients and likely future patients. Significant applications are developed to track data of confirmed, recovered, and death cases. In this work, we propose a new feature selection method based on a new way of hybridization between filter and wrapper methods. The proposed approach is expected to achieve high classification accuracy with a small feature subset. Specifically, the main contribution of this work is a four steps-based approach organized as follows: First, we remove consecutively duplicate and constant features. Then, we select the highest-ranked feature with Mutual Information. In the last step, we run the ‘Backward Feature Elimination’ algorithm to delete features from the active subset until a stopping criterion based on the degradation of classification performance is met. We applied the proposed approach to a COVID-19 dataset to test its ability to find the relevant feature for characterizing the disease, such as new cases infected with the virus, people vaccinated, and the number of deaths, to better assess the situation. For evaluation purposes, experiments are conducted at the first stage on the COVID-19 dataset, then on six benchmark datasets that have a high dimensional and large size. The method performance is tracked and measured on these datasets and a comparison with many approaches is provided.

Adaptive CNN filter pruning using global importance metric

The depth and width of CNNs have increased over the years so as to learn a better representation of the input–output mapping of a dataset. However, a significant amount of redundancy exists among different convolutional kernels. Several methods on pruning suggest that trimming redundant parameters can produce compact structures with minor degradation in classification performance. Existing pruning methods reduce the number of filters at a uniform rate (i.e. pruning same percentage of filters from each layer) in every convolutional layer, which is suboptimal. In this paper, we conduct experiments to observe the sensitivity of each and every filter towards the final performance of the neural network. The essence of comparing filter importance on a global scale and subsequently pruning the neural network adaptively, is highlighted for the first time in this paper. Based on our observations, we propose a novel method named ‘Global Filter Importance based Adaptive Pruning (GFI-AP)’ that assigns importance scores to all filters based on how the network learns the input–output mapping of a dataset, which can then be compared across all the other convolutional filters. Our results show that non-uniform pruning achieves better compression as compared to uniform pruning. We demonstrate that GFI-AP significantly decreases the number of FLOPs (floating point operations) of VGG and ResNet networks in ImageNet and CIFAR datasets, without substantial drop in classification accuracy. GFI-AP reduces more number of FLOPs compared to existing pruning methods, for example, the ResNet50 variant of GFI-AP provides an additional 11% reduction in FLOPs over Taylor-FO-BN-72% while achieving higher accuracy.