Decision Boundary Of Classifier Research Articles

Class incremental learning with KL constraint and multi-strategy exemplar selection for classification based on MMFA model

Class incremental learning (CIL) can learn new classes continuously by updating the model rather than retraining a model from scratch with all seen classes like traditional offline learning, therefore, CIL is more suitable for classification in dynamic environments, where new classes are captured progressively. However, the key knowledge of old classes will be lost due to the update mode of CIL, leading to the catastrophic forgetting (CF) problem. In this paper, a novel CIL method with Kullback-Leibler constraint and multi-strategy exemplar selection (CIL-KLMES) is proposed for classification based on max-margin factor analysis (MMFA) model. To handle the CF problem, CIL-KLMES imposes a Kullback-Leibler (KL) divergence term on the important parameters when updating the model to restrict the parameters' distribution to be similar, thus preventing the updated model from deviating too much from the previous model and preserving the knowledge of old classes. Moreover, CIL-KLMES selects a few representative exemplars from old classes based on the robust description of data distribution and classification decision boundary. By replaying representative exemplars to update the model together with new class data, the key knowledge of old classes can be further preserved. Therefore, the CF problem can be alleviated sufficiently. Experimental results demonstrate the effectiveness of CIL-KLMES.

Genetically optimised SMOTE-based adversarial discriminative domain adaptation for rotor fault diagnosis at variable operating conditions

Abstract For safety, reliability, and uninterrupted output of gas turbines, aviation engines, power-generating equipment, pumps, gears, compressors etc., rotor mass imbalance must be detected and diagnosed to avoid catastrophic failure. Industry 4.0 relies on predictive digital maintenance and deep learning-based convolutional neural network (CNN), which predicts defects but fails if the operating conditions change. Research studies in various fields indicate that the domain shift issue occurs due to source and target samples being from different domains, which reduces prediction capability. Moreover, research studies are scarce in examining prediction capability under varying operating speeds for rotor mass imbalance. Hence, this research proposes the adversarial discriminative domain adaptation (ADDA) technique which predicts machine failures under various operational conditions. The efficacy of ADDA has been explored by introducing 1D-CNN as a source and a target encoder inside ADDA's architecture to take advantage of CNN's feature extraction capability. Further, this research effectively tackles CNN's inherent issues of overfitting and hyperparameters value selection. Furthermore, The real-world scenario has more healthy samples than fault condition samples, causing a multiclass imbalance in sample data, which affects the classification decision boundary and causes biased prediction. Hence, the proposed methodology first addresses the class imbalance through synthetic minority oversampling (SMOTE), then genetic algorithm optimizes 1D-CNN's hyperparameters, and the effective dropout layer positioning solves the overfitting. Finally, the deep learning-based SMOTE_ADDA_GO-1D-CNN decreases domain discrepancy with adversarial discriminative domain adaptation. The proposed methodology's efficacy has been explored through F1-Score, which is used as multiclass evaluation metrics, and it has been benchmarked against standard machine learning and deep learning algorithms. The test results of the proposed methodology surpassed all of them with maximum prediction accuracy. Thus, this study contributes to rotor mass imbalance detection and diagnosis for multiclass imbalanced data under varying operational conditions by successfully overcoming potential challenges during fault prediction.

Sensing Pre- and Post-Ecdysis of Tropical Rock Lobsters, Panulirus ornatus, Using a Low-Cost Novel Spectral Camera

Tropical rock lobsters (Panulirus ornatus) are a highly cannibalistic species with intermoult animals predominantly attacking animals during ecdysis (moulting). Rapid, positive characterisation of pre-ecdysis lobsters may open a pathway to disrupt cannibalism. Ecdysial suture line development is considered for pre-ecdysis recognition with suture line definition compared for intermoult and pre-ecdysis lobsters emerged and immerged, using white, near ultraviolet (365 nm), near infrared (850 nm), and specialty SFH 4737 broadband IR LEDs against a reference of intermoult lobsters with no suture line development. Difficulties in acquiring suture line images prompted research into pre-ecdysis characterisation from the lobster’s dorsal carapace, due to its accessibility through a culture vessel’s surface. In this study, a novel low-cost spectral camera was developed by coordinating an IMX219 image sensor, an AS7265x spectral sensor, and four SFH 4737 broadband infrared LEDs through a single-board computer. Images and spectral data from the lobster’s dorsal carapace were acquired from intermoult, pre-ecdysis, and post-ecdysis lobsters. For the first time, suture line definition was found to be enhanced under 850 nm, 365 nm, and SFH 4737 LEDs for immerged lobsters, while the 850 nm LED achieved the best suture line definition of emerged lobsters. Although the spectral camera was unable to characterise pre-ecdysis, its development was validated when a least squares regression for binary classification decision boundary successfully separated 86.7% of post-ecdysis lobsters. Achieving post-ecdysis characterisation is the first time the dorsal carapace surface has been used to characterise a moult stage for palinurid lobsters.

A clustering multi-stage training transfer learning framework for cross simulation domain and experimental domain fault diagnosis

Deep learning methods have demonstrated remarkable achievements in the field of fault diagnosis for rotating machinery. However, their effectiveness heavily relies on high-quality labeled samples, which presents a significant challenge owing to the limited availability of such data in engineering applications. To address this realistic issue, we propose a novel simulation-driven transfer learning model called the clustering multi-stage training transfer learning framework (CMSTL) for fault diagnosis of rolling bearings. The fundamental concept of the proposed method is to utilize simulation data as a substitute for labeled actual device data and integrate the suggested clustering learning and multistage training strategies to extract domain-independent and fault-discriminative features from simulation and experimental domains. Specifically, the clustering learning strategy is embedded into the CMSTL model to encourage the feature extractor to acquire distinguishable features associated with different categories while eliminating domain-specific knowledge, which enables samples near the classification decision boundary to cluster towards their respective clustering centers. Additionally, the proposed multistage learning strategy leverages the model trained with a certain level of accuracy in the first stage to annotate actual device samples, thereby enhancing both the precision of the pseudo-label for real data and the overall training stability of the model. The effectiveness and superiority of the proposed method were validated using both artificially damaged and run-to-failure datasets. The comparative analysis results demonstrate that the CMSTL method exhibits a minimum 2.2% improvement in fault diagnosis accuracy and enhances the clustering capability compared to other advanced transfer learning fault diagnosis methods.

EBBE-Text: Explaining Neural Networks by Exploring Text Classification Decision Boundaries.

While neural networks (NN) have been successfully applied to many NLP tasks, the way they function is often difficult to interpret. In this article, we focus on binary text classification via NNs and propose a new tool, which includes a visualization of the decision boundary and the distances of data elements to this boundary. This tool increases the interpretability of NN. Our approach uses two innovative views: (1) an overview of the text representation space and (2) a local view allowing data exploration around the decision boundary for various localities of this representation space. These views are integrated into a visual platform, EBBE-Text, which also contains state-of-the-art visualizations of NN representation spaces and several kinds of information obtained from the classification process. The various views are linked through numerous interactive functionalities that enable easy exploration of texts and classification results via the various complementary views. A user study shows the effectiveness of the visual encoding and a case study illustrates the benefits of using our tool for the analysis of the classifications obtained with several recent NNs and two datasets.

Source-free unsupervised domain adaptation via bi-classifier confidence score weighting

Source-free unsupervised domain adaptation (SFUDA) seeks to adapt a pre-trained model from the source domain to the unlabeled target domain without using any source domain data. Some existing methods focus on learning transferable representations with only one classifier and cannot exploit full knowledge of the target domain. Therefore, they are unable to obtain a clear classification decision boundary between the source domain and the target domain. Therefore, we propose a Bi-classifier Confidence Score Weighting (BCSW) approach for addressing these issues. We first introduce a novel confidence score for sample weighting adaptation to boost the performance of the target model. Additionally, we exploit the sample mixup strategy to adequately utilize target domain knowledge to form better clusters. We also further design bi-classifier and adopt classifier disparity distance(CDD) to train the target model to effectively push cross-domain distributions close. Experiments show that our method achieves results equal to or better than the most advanced methods on standard benchmarks.

Wear identification of end mills based on a feature-weighted convolutional neural network under unbalanced samples

The number of samples with various wear states gathered in processing is unbalanced because the wear rate of end mills fluctuates nonlinearly. There are still some limitations to the current approaches for identifying the end mill wear condition in this situation. It is primarily manifested in the dominance of the majority class samples in forming the classification decision boundary. The minority class samples and the samples with a more significant influence on the decision boundary in the majority class samples are closer and difficult to separate, making it challenging to identify the wear state of minority classes. Herein, a deep feature-weighted convolutional neural network (DFWCNN)-based end mill wear state identification approach is proposed to overcome the above limitations. First, the balance influence factor selectively reduces the weight of the samples that significantly influence the decision boundary to lessen the majority classes' dominance on the classification decision boundary. Then, the variance influence factor reduces the intraclass distance to lessen confusion between the samples at the classification decision boundary. The experiments conducted herein have proved that the proposed approach improves the identification accuracy of samples with various wear states, particularly accelerated wear state samples, in the case of sample imbalance.

Curiosity-Driven Class-Incremental Learning via Adaptive Sample Selection

Modern artificial intelligence systems require class-incremental learning while suffering from catastrophic forgetting in many real-world applications. Due to the missing knowledge of past data, performance substantially degrades. Recent methods often used knowledge distillation and bias correction to avoid catastrophic forgetting caused by cognitive bias. However, since these methods mainly learn all samples indiscriminately, the model is hard to learn what it truly needs from the data stream to balance the new and old knowledge, leading to inevitable forgetting. Instead of considering each sample indiscriminately, the model should learn from its curious samples automatically. To tackle this problem, we propose a curiosity-driven class-incremental learning approach via adaptive sample selection for learning a more generalized model with fewer ineffective updates. Specifically, our method quantifies the model’s curiosity in each sample by two properties: uncertainty and novelty. Our model learns informative samples selectively during training utilizing the proposed uncertainty property, which benefits the classification decision boundary. In light of the imbalanced data, a novelty property is used to selectively optimize the model by employing dissimilar samples, endowing it with more robustness and less cognitive bias. Our method successfully reduces catastrophic forgetting and can be flexibly incorporated with other techniques. Extensive experiments and in-depth analysis on the CIFAR-100, Tiny-ImageNet and Caltech-101 datasets show that our approach outperforms competing methods for class-incremental learning in terms of preventing catastrophic forgetting.

Machine learning interpretability for a stress scenario generation in credit scoring based on counterfactuals

To boost the application of machine learning (ML) techniques for credit scoring models, the blackbox problem should be addressed. The primary aim of this paper is to propose a measure based on counterfactuals to evaluate the interpretability of a ML credit scoring technique. Counterfactuals assist with understanding the model with regard to the classification decision boundaries and evaluate model robustness. The second contribution is the development of a data perturbation technique to generate a stress scenario.We apply these two proposals to a dataset on UK unsecured personal loans to compare logistic regression and stochastic gradient boosting (SBG). We show that training a blackbox model (SGB) as conditioned on our data perturbation technique can provide insight into model performance under stressed scenarios. The empirical results show that our interpretability measure is able to capture the classification decision boundary, unlike AUC and the classification accuracy widely used in the banking sector.

Early- and in-season crop type mapping without current-year ground truth: Generating labels from historical information via a topology-based approach

Land cover classification in remote sensing is often faced with the challenge of limited ground truth labels. Incorporating historical ground information has the potential to significantly lower the expensive cost associated with collecting ground truth and, more importantly, enable early- and in-season mapping that is helpful to many pre-harvest decisions. In this study, we propose a new approach that can effectively transfer knowledge about the topology (i.e. relative position) of different crop types in the spectral feature space (e.g. the histogram of SWIR1 vs RDEG1 bands) to generate labels, thereby supporting crop classification in a different year. Importantly, our approach does not attempt to transfer classification decision boundaries that are susceptible to inter-annual variations of weather and management, but relies on the more robust and shift-invariant topology information. We tested this approach for mapping corn/soybeans in the US Midwest, paddy rice/corn/soybeans in Northeast China and multiple crops in Northern France using Landsat-8 and Sentinel-2 data. Results show that our approach automatically generates high-quality labels for crops in the target year immediately after each image becomes available. Based on these generated labels from our approach, the subsequent crop type mapping using a random forest classifier can reach the F1 score as high as 0.887 for corn as early as the silking stage and 0.851 for soybean as early as the flowering stage and the overall accuracy of 0.873 in the test state of Iowa. In Northeast China, F1 scores of paddy rice, corn and soybeans and the overall accuracy can exceed 0.85 two and half months ahead of harvest. In the Hauts-de-France region, the OA of multiple crop mapping could reach 0.837 based on labels generated from our approach. Overall, these results highlight the unique advantages of our approach in transferring historical knowledge and maximizing the timeliness of crop maps. Our approach supports a general paradigm shift towards learning transferrable and generalizable knowledge to facilitate land cover classification.

SEGCN-DCR: A syntax-enhanced event detection framework with decoupled classification rebalance

Event Detection (ED) is a pivotal sub-task of Event Extraction(EE). It aims to locate triggers and categorize them into specific event types. Recent researches on ED have shown that graph convolutional neural networks with syntactic information can achieve advanced performance. However, these methods ignore the implicit importance score of tokens. This will weaken their ability of identifying trigger words. In addition, due to the long-tailed distribution in the corpus, previous methods perform poorly on sparsely labeled trigger words and are prone to overfitting on densely labeled ones. In this paper, we propose a Syntax-Enhanced GCN framework with a Decoupled Classification Rebalance mechanism (SEGCN-DCR) to address the above issues. Specifically, we exploit a tree-structured module based on dependency structure to reduce the noise by capturing global hierarchical syntactic information, and DCR mechanism to rescale the classifier weights, which makes classifier decision boundaries more reasonable. Experiments on benchmark ACE2005 show that the proposed method acquires state-of-the-art performance.

An Unsupervised Domain Adaptation Method Towards Multi-Level Features and Decision Boundaries for Cross-Scene Hyperspectral Image Classification

An Unsupervised Domain Adaptation Method Towards Multi-Level Features and Decision Boundaries for Cross-Scene Hyperspectral Image Classification

A Bootstrapping Solution for Effective Interpretation of Transformer Winding Frequency Response

The current interpretation procedure for the transformer frequency response analysis (FRA) is practically based on either subjective visual inspection or an observed value of a statistical indicator (SIn) programed in commercial software. However, both solutions are unable to provide very detailed information on transformer conditions in the industry. The former approach highly depends on the experience of the technical personnel, while the latter suffers from uncertainties of SIn-based decision boundaries for classification. This study has specifically focused and introduced a new comprehensive FRA data interpretation and classification method. Evaluation of the proposed technique is performed by emulation of winding short-circuit faults in transformers via a parallel connected rheostat. The method is able to categorize the operational mode of the transformer into the normal, suspicious, and critical classes based on the cumulative effect of 12 frequently used SIns and then provide a confidence level (CL) associated with classification to support operators for making a precise decision. Bootstrap sampling technique is introduced for the first time in FRA data evaluation to simulate the effect of many experiments <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">in silico</i> and overcome the difficulty of limited available measured data. Particularly, in a novel approach, a large set of SIn-specific decision boundaries is established by bootstrap sampling of the real experimentally obtained boundaries. In addition, a new transformer condition assessment method that employs the cumulative effect of individual SIns is developed and examined. Classification results and the CLs in FRA evaluation have a substantial practical significance and will support on-site utility managers or engineers for further diagnosis and maintenance.

Convolutional feature extraction for process monitoring using ultrasonic sensors

Ultrasonic sensors are a low-cost and in-line technique and can be combined with machine learning for industrial process monitoring. However, training accurate machine learning models for process monitoring using sensor data is dependant on the feature selection methodology. This paper compares a convolutional feature extraction method to a traditional, coarse feature engineering approach. The convolutional method uses filter weights pre-trained on an auxiliary task to classify ultrasonic waveform dataset membership using previously obtained sensor data. The filter weights are used to extract features from the ultrasonic waveform. Principal component analysis is then applied to produce five principal components to be input into long short-term memory neural networks. The two approaches are compared on fermentation, mixing and cleaning datasets monitored using ultrasonic sensors. Overall, the convolutional feature method produced more informative waveform features than the coarse feature engineering approach, achieving higher model accuracy for datasets requiring substantial waveform information and for 65% of tasks overall. Multi-task learning also improved feature trajectory learning but led to reduced model accuracy for data points far from the classification decision boundaries. This can be overcome by further optimisation of neural network hyperparameters, though at increased model development time. Once trained, the convolutional feature extraction approach is a fast and convenient way of producing high quality features from ultrasonic waveforms using convolutional neural networks with little training data.

Rule-based space characterization for rumour detection in health

Last decades have witnessed a radical change in the way information spreads. Social networks provide a constantly updated pool of news to the end-users but the absence of systematic control and moderation of the posts easily leads to spread unverified news with an instrumental value and likely to be dangerous, which are referred to as rumours. To tackle this issue various systems for automatic rumour detection among conversations, i.e. an aggregated set of posts, have been recently presented in the literature. However, few efforts have been directed towards rumour detection at the level of single posts (micro-level), which is the challenging scenario that we tackle in this work. Moving at a finer scale is an urgent need since both rumour and non-rumour posts are included in the same conversation. Here the rumour detection issue is addressed presenting a novel feature selection approach, which characterizes the feature space aiming at minimizing samples in unreliable configurations. This approach is compared with other state-of-the-art methods using a pool of different learning algorithms on two health-related Twitter datasets, labelled at the micro-level. Our proposal yields promising results: it outperforms other feature selection approaches with a best accuracy of 96.8% and enhances the performance of our previous work up to 5%. These findings prove the potential of the feature selection method introduced, which gives access to samples distribution in the feature space, providing privileged information for the construction of the classifier decision boundaries. Nonetheless they also bring a step forward the micro-level rumour detection analysis.

Deep Open Intent Classification with Adaptive Decision Boundary

Open intent classification is a challenging task in dialogue systems. On the one hand, it should ensure the quality of known intent identification. On the other hand, it needs to detect the open (unknown) intent without prior knowledge. Current models are limited in finding the appropriate decision boundary to balance the performances of both known intents and the open intent. In this paper, we propose a post-processing method to learn the adaptive decision boundary (ADB) for open intent classification. We first utilize the labeled known intent samples to pre-train the model. Then, we automatically learn the adaptive spherical decision boundary for each known class with the aid of well-trained features. Specifically, we propose a new loss function to balance both the empirical risk and the open space risk. Our method does not need open intent samples and is free from modifying the model architecture. Moreover, our approach is surprisingly insensitive with less labeled data and fewer known intents. Extensive experiments on three benchmark datasets show that our method yields significant improvements compared with the state-of-the-art methods.

Learning decision boundaries for cone penetration test classification

AbstractIn geotechnical field investigations, cone penetration tests (CPT) are increasingly used for ground characterization of fine‐grained soils. Test results are different parameters that are typically visualized in CPT based data interpretation charts. In this paper we propose a novel methodology which is based on supervised machine learning that permits a redefinition of the boundaries within these charts to account for unique soil conditions. We train ensembles of randomly generated artificial neural networks to classify six soil types based on a database of hundreds of CPT tests from Austria and Norway. After training we combine the multiple unique solutions for this classification problem and visualize the new decision boundaries in between the soil types. The generated boundaries between soil types are comprehensible and are a step towards automatically adjusted CPT interpretation charts for specific local conditions.

Meta-reweighted Regularization for Unsupervised Domain Adaptation

Unsupervised domain adaptation enables knowledge transfer from a labeled source domain to an unlabeled target domain by reducing the cross-domain distribution discrepancy, and the adversarial learning based paradigm has achieved remarkable success. On top of this, recent works seeks to further regularize the classification decision boundary via self-training to learn target adaptive classifier with pseudo-labeled target samples. However, since pseudo labels are inevitably noisy, most of prior methods focus on manually designing elaborate target selection algorithms or optimization objectives. Different from them, we propose a meta-learning based target-reweighting regularization algorithm called MetaReg. Specifically, MetaReg is motivated by the intuition that an ideal target classifier trained on correct target pseudo labels should make small classification errors on target-like source samples. Therefore, we explicitly define a meta reweighting problem that aims to find optimal weights for different samples by minimizing the classification loss on a class-balanced set consisting of source samples that are most similar to target ones. The optimization problem is solved efficiently with a simplified approximation technique. As a result, the automatically learned optimal weights are utilized to reweight pseudo-labeled target samples and regularize the model learning. Comprehensive experiments verify that MetaReg outperforms the non-regularized UDA counterparts with state-of-the-art performance.

Image Descriptors for Weakly Annotated Histopathological Breast Cancer Data

Introduction: Cancerous Tissue Recognition (CTR) methodologies are continuously integrating advancements at the forefront of machine learning and computer vision, providing a variety of inference schemes for histopathological data. Histopathological data, in most cases, come in the form of high-resolution images, and thus methodologies operating at the patch level are more computationally attractive. Such methodologies capitalize on pixel level annotations (tissue delineations) from expert pathologists, which are then used to derive labels at the patch level. In this work, we envision a digital connected health system that augments the capabilities of the clinicians by providing powerful feature descriptors that may describe malignant regions.Material and Methods: We start with a patch level descriptor, termed Covariance-Kernel Descriptor (CKD), capable of compactly describing tissue architectures associated with carcinomas. To leverage the recognition capability of the CKDs to larger slide regions, we resort to a multiple instance learning framework. In that direction, we derive the Weakly Annotated Image Descriptor (WAID) as the parameters of classifier decision boundaries in a Multiple Instance Learning framework. The WAID is computed on bags of patches corresponding to larger image regions for which binary labels (malignant vs. benign) are provided, thus obviating the necessity for tissue delineations.Results: The CKD was seen to outperform all the considered descriptors, reaching classification accuracy (ACC) of 92.83%. and area under the curve (AUC) of 0.98. The CKD captures higher order correlations between features and was shown to achieve superior performance against a large collection of computer vision features on a private breast cancer dataset. The WAID outperform all other descriptors on the Breast Cancer Histopathological database (BreakHis) where correctly classified malignant (CCM) instances reached 91.27 and 92.00% at the patient and image level, respectively, without resorting to a deep learning scheme achieves state-of-the-art performance.Discussion: Our proposed derivation of the CKD and WAID can help medical experts accomplish their work accurately and faster than the current state-of-the-art.

Reaction times predict dynamic brain representations measured with MEG for only some object categorisation tasks

Behavioural categorisation reaction times (RTs) provide a useful way to link behaviour to brain representations measured with neuroimaging. In this framework, objects are assumed to be represented in a multidimensional activation space, with the distances between object representations indicating their degree of neural similarity. Faster RTs have been reported to correlate with greater distances from a classification decision boundary for animacy. Objects inherently belong to more than one category, yet it is not known whether the RT-distance relationship, and its evolution over the time-course of the neural response, is similar across different categories. Here we used magnetoencephalography (MEG) to address this question. Our stimuli included typically animate and inanimate objects, as well as more ambiguous examples (i.e., robots and toys). We conducted four semantic categorisation tasks on the same stimulus set assessing animacy, living, moving, and human-similarity concepts, and linked the categorisation RTs to MEG time-series decoding data. Our results show a sustained RT-distance relationship throughout the time course of object processing for not only animacy, but also categorisation according to human-similarity. Interestingly, this sustained RT-distance relationship was not observed for the living and moving category organisations, despite comparable classification accuracy of the MEG data across all four category organisations. Our findings show that behavioural RTs predict representational distance for an organisational principle other than animacy, however further research is needed to determine why this relationship is observed only for some category organisations and not others.