Decision Boundary Of Classifier Research Articles

Learning decision boundaries for cone penetration test classification

AbstractIn geotechnical field investigations, cone penetration tests (CPT) are increasingly used for ground characterization of fine‐grained soils. Test results are different parameters that are typically visualized in CPT based data interpretation charts. In this paper we propose a novel methodology which is based on supervised machine learning that permits a redefinition of the boundaries within these charts to account for unique soil conditions. We train ensembles of randomly generated artificial neural networks to classify six soil types based on a database of hundreds of CPT tests from Austria and Norway. After training we combine the multiple unique solutions for this classification problem and visualize the new decision boundaries in between the soil types. The generated boundaries between soil types are comprehensible and are a step towards automatically adjusted CPT interpretation charts for specific local conditions.

Meta-reweighted Regularization for Unsupervised Domain Adaptation

Unsupervised domain adaptation enables knowledge transfer from a labeled source domain to an unlabeled target domain by reducing the cross-domain distribution discrepancy, and the adversarial learning based paradigm has achieved remarkable success. On top of this, recent works seeks to further regularize the classification decision boundary via self-training to learn target adaptive classifier with pseudo-labeled target samples. However, since pseudo labels are inevitably noisy, most of prior methods focus on manually designing elaborate target selection algorithms or optimization objectives. Different from them, we propose a meta-learning based target-reweighting regularization algorithm called MetaReg. Specifically, MetaReg is motivated by the intuition that an ideal target classifier trained on correct target pseudo labels should make small classification errors on target-like source samples. Therefore, we explicitly define a meta reweighting problem that aims to find optimal weights for different samples by minimizing the classification loss on a class-balanced set consisting of source samples that are most similar to target ones. The optimization problem is solved efficiently with a simplified approximation technique. As a result, the automatically learned optimal weights are utilized to reweight pseudo-labeled target samples and regularize the model learning. Comprehensive experiments verify that MetaReg outperforms the non-regularized UDA counterparts with state-of-the-art performance.

Image Descriptors for Weakly Annotated Histopathological Breast Cancer Data

Introduction: Cancerous Tissue Recognition (CTR) methodologies are continuously integrating advancements at the forefront of machine learning and computer vision, providing a variety of inference schemes for histopathological data. Histopathological data, in most cases, come in the form of high-resolution images, and thus methodologies operating at the patch level are more computationally attractive. Such methodologies capitalize on pixel level annotations (tissue delineations) from expert pathologists, which are then used to derive labels at the patch level. In this work, we envision a digital connected health system that augments the capabilities of the clinicians by providing powerful feature descriptors that may describe malignant regions.Material and Methods: We start with a patch level descriptor, termed Covariance-Kernel Descriptor (CKD), capable of compactly describing tissue architectures associated with carcinomas. To leverage the recognition capability of the CKDs to larger slide regions, we resort to a multiple instance learning framework. In that direction, we derive the Weakly Annotated Image Descriptor (WAID) as the parameters of classifier decision boundaries in a Multiple Instance Learning framework. The WAID is computed on bags of patches corresponding to larger image regions for which binary labels (malignant vs. benign) are provided, thus obviating the necessity for tissue delineations.Results: The CKD was seen to outperform all the considered descriptors, reaching classification accuracy (ACC) of 92.83%. and area under the curve (AUC) of 0.98. The CKD captures higher order correlations between features and was shown to achieve superior performance against a large collection of computer vision features on a private breast cancer dataset. The WAID outperform all other descriptors on the Breast Cancer Histopathological database (BreakHis) where correctly classified malignant (CCM) instances reached 91.27 and 92.00% at the patient and image level, respectively, without resorting to a deep learning scheme achieves state-of-the-art performance.Discussion: Our proposed derivation of the CKD and WAID can help medical experts accomplish their work accurately and faster than the current state-of-the-art.

Reaction times predict dynamic brain representations measured with MEG for only some object categorisation tasks

Behavioural categorisation reaction times (RTs) provide a useful way to link behaviour to brain representations measured with neuroimaging. In this framework, objects are assumed to be represented in a multidimensional activation space, with the distances between object representations indicating their degree of neural similarity. Faster RTs have been reported to correlate with greater distances from a classification decision boundary for animacy. Objects inherently belong to more than one category, yet it is not known whether the RT-distance relationship, and its evolution over the time-course of the neural response, is similar across different categories. Here we used magnetoencephalography (MEG) to address this question. Our stimuli included typically animate and inanimate objects, as well as more ambiguous examples (i.e., robots and toys). We conducted four semantic categorisation tasks on the same stimulus set assessing animacy, living, moving, and human-similarity concepts, and linked the categorisation RTs to MEG time-series decoding data. Our results show a sustained RT-distance relationship throughout the time course of object processing for not only animacy, but also categorisation according to human-similarity. Interestingly, this sustained RT-distance relationship was not observed for the living and moving category organisations, despite comparable classification accuracy of the MEG data across all four category organisations. Our findings show that behavioural RTs predict representational distance for an organisational principle other than animacy, however further research is needed to determine why this relationship is observed only for some category organisations and not others.

Semi-supervised learning using adversarial training with good and bad samples

In this work, we investigate semi-supervised learning (SSL) for image classification using adversarial training. Previous results have illustrated that generative adversarial networks (GANs) can be used for multiple purposes. Triple-GAN, which aims to jointly optimize model components by incorporating three players, generates suitable image-label pairs to compensate for the lack of labeled data in SSL with improved benchmark performance. Conversely, Bad (or complementary) GAN, optimizes generation to produce complementary data-label pairs and force a classifier's decision boundary to lie between data manifolds. Although it generally outperforms Triple-GAN, Bad GAN is highly sensitive to the amount of labeled data used for training. Unifying these two approaches, we present unified-GAN (UGAN), a novel framework that enables a classifier to simultaneously learn from both good and bad samples through adversarial training. We perform extensive experiments on various datasets and demonstrate that UGAN: 1) achieves state-of-the-art performance among other deep generative models, and 2) is robust to variations in the amount of labeled data used for training.

Unsupervised Domain Adaptation via Discriminative Classes-Center Feature Learning in Adversarial Network

Adversarial learning has achieved remarkable advance in learning transferable representations across different domains. Generally, previous works are mainly devoted to reducing domain shift between labeled source data and unlabeled target data by extracting domain-invariant features. However, these adversarial methods rarely consider task-specific decision boundaries among classes, causing classification performance degradation in cross domain tasks. In this paper, we propose a novel approach for the task of unsupervised domain adaptation via discriminative classes-center feature learning in adversarial network (C2FAN), which concentrates on learning domain-invariant representation and paying close attention to classification decision boundary simultaneously to improve the ability of transferable knowledge across different domains. C2FAN consists of a feature extractor, a classifier and a discriminator. Firstly, for reducing domain gaps between source and target domains in the feature extractor, we propose to utilize a conditional adversarial learning module to extract domain-invariant feature and improve discriminability of the classifier simultaneously. Further, we present a high-efficiency layer normalization module to reduce domain shift existing in the classifier. Secondly, we design a discriminative classes-center feature learning module in the classifier to diminish the distribution distance of the same-class samples so that the decision boundary can distinguish different classes easily, which can reduce the misclassification on target samples. What’s more, C2FAN is an effective yet considerable simple approach which can be embedded into current domain adaptation approaches conveniently. Extensive experiments demonstrate that our proposed model achieves satisfactory results on some standard domain adaptation benchmarks.

Unsupervised image translation using adversarial networks for improved plant disease recognition

Acquisition of data in task-specific applications of machine learning like plant disease recognition is a costly endeavor owing to the requirements of professional human diligence and time constraints. In this paper, we present a pipeline that uses GANs in an unsupervised image translation environment to improve learning with respect to the data distribution in a plant disease dataset, reducing the partiality introduced by acute class imbalance and hence shifting the classification decision boundary towards better performance. The empirical analysis of our method is demonstrated on a limited dataset of 2789 tomato plant disease images, highly corrupted with an imbalance in the 9 disease categories. First, we extend the state of the art for the GAN-based image-to-image translation method by enhancing the perceptual quality of the generated images and preserving the semantics. We introduce AR-GAN, where in addition to the adversarial loss, our synthetic image generator optimizes on Activation Reconstruction loss (ARL) function that optimizes feature activations against the natural image. We present visually more compelling synthetic images in comparison to most prominent existing models and evaluate the performance of our GAN framework in terms of various datasets and metrics. Second, we evaluate the performance of a baseline convolutional neural network classifier for improved recognition using the resulting synthetic samples to augment our training set and compare it with the classical data augmentation scheme. We observe a significant improvement in classification accuracy (+5.2%) using generated synthetic samples as compared to (+0.8%) increase using classic augmentation in an equal class distribution environment.

Using neural distance to predict reaction time for categorizing the animacy, shape, and abstract properties of objects

A large number of neuroimaging studies have shown that information about object category can be decoded from regions of the ventral visual pathway. One question is how this information might be functionally exploited in the brain. In an attempt to help answer this question, some studies have adopted a neural distance-to-bound approach, and shown that distance to a classifier decision boundary through neural activation space can be used to predict reaction times (RT) on animacy categorization tasks. However, these experiments have not controlled for possible visual confounds, such as shape, in their stimulus design. In the present study we sought to determine whether, when animacy and shape properties are orthogonal, neural distance in low- and high-level visual cortex would predict categorization RTs, and whether a combination of animacy and shape distance might predict RTs when categories crisscrossed the two stimulus dimensions, and so were not linearly separable. In line with previous results, we found that RTs correlated with neural distance, but only for animate stimuli, with similar, though weaker, asymmetric effects for the shape and crisscrossing tasks. Taken together, these results suggest there is potential to expand the neural distance-to-bound approach to other divisions beyond animacy and object category.

Practical Implementation of Memristor-Based Threshold Logic Gates

Current advances in emerging memory technologies enable novel and unconventional computing architectures for high-performance and low-power electronic systems, capable of carrying out massively parallel operations at the edge. One emerging technology, ReRAM, also known to belong in the family of memristors (memory resistors), is gathering attention due to its attractive features for logic and in-memory computing and benefits which follow from its technological attributes, such as nanoscale dimensions, low-power operation, and multi-state programming. At the same time, the design with CMOS is quickly reaching its physical and functional limitations, and further research toward novel logic families, such as threshold logic gates (TLGs), is scoped. In this paper, we introduce a physical implementation of a memristor-based current-mode TLG (MCMTLG) circuit and validate its design and operation through multiple experimental setups. We demonstrate two-input, three-input, and four-input MCMTLG configurations and showcase their reconfiguration capability. This is achieved by varying memristive weights arbitrarily for shaping the classification decision boundary, thus showing a promise as an alternative hardware-friendly implementation of artificial neural networks. Through the employment of real memristor devices as the equivalent of synaptic weights in TLGs, we are realizing components that can be used toward an in silico classifier.

Waveform prototype-based feature learning for automatic detection of the early repolarization pattern in ECG signals

Objective: Our aim was to develop an automated detection method, for prescreening purposes, of early repolarization (ER) pattern with slur/notch configuration in electrocardiogram (ECG) signals using a waveform prototype-based feature vector for supervised classification. Approach: The feature vectors consist of fragments of the ECG signal where the ER pattern is located, instead of abstract descriptive variables of ECG waveforms. The tested classifiers included linear discriminant analysis, k-nearest neighbor algorithm, and support vector machine (SVM). Main results: SVM showed the best performance in Friedman tests in our test data including 5676 subjects representing 45 408 leads. Accuracies of the different classifiers showed results well over 90%, indicating that the waveform prototype-based feature vector is an effective representation of the differences between ECG signals with and without the ER pattern. The accuracy of inferior ER was 92.74% and 92.21% for lateral ER. The sensitivity achieved was 91.80% and specificity was 92.73%. Significance: The algorithm presented here showed good performance results, indicating that it could be used as a prescreening tool of ER, and it provides an additional identification of critical cases based on the distances to the classifier decision boundary, which are close to the 0.1 mV threshold and are difficult to label.

Categorizing natural color distributions

The natural objects that we are surrounded with virtually always contain many different shades of color, yet the visual system usually categorizes them into a single color category. We examined various image statistics and their role in categorizing the color of leaves. Our subjects categorized photographs of autumn leaves and versions that were manipulated, including: randomly repositioned pixels, leaves uniformly colored with their mean color, leaves that were made by reflecting the original leaves' chromaticity distribution about their mean (“flipped leaves”), and simple patches colored with the mean colors of the original leaves. We trained a linear classifier with a set of image statistics in order to predict the category that each object was assigned to. Our results show that the mean hue of an object is highly predictive of the natural object's color category (>90% accuracy) and observers’ choices are consistent with their use of unique yellow as a decision boundary for classification. The flipped leaves produced consistent changes in color categorization that are possibly explained by an interaction between the color distributions and the texture of the leaves.

SetSVM: An Approach to Set Classification in Nuclei-Based Cancer Detection.

Due to the importance of nuclear structure in cancer diagnosis, several predictive models have been described for diagnosing a wide variety of cancers based on nuclear morphology. In many computer-aided diagnosis (CAD) systems, cancer detection tasks can be generally formulated as set classification problems, which can not be directly solved by classifying single instances. In this paper, we propose a novel set classification approach SetSVM to build a predictive model by considering any nuclei set as a whole without specific assumptions. SetSVM features highly discriminative power in cancer detection challenges in the sense that it not only optimizes the classifier decision boundary but also transfers discriminative information to set representation learning. During model training, these two processes are unified in the support vector machine (SVM) maximum separation margin problem. Experiment results show that SetSVM provides significant improvements compared with five commonly used approaches in cancer detection tasks utilizing 260 patients in total across three different cancer types, namely, thyroid cancer, liver cancer, and melanoma. In addition, we show that SetSVM enables visual interpretation of discriminative nuclear characteristics representing the nuclei set. These features make SetSVM a potentially practical tool in building accurate and interpretable CAD systems for cancer detection.

Asymmetric Compression of Representational Space for Object Animacy Categorization under Degraded Viewing Conditions.

Animacy is a robust organizing principle among object category representations in the human brain. Using multivariate pattern analysis methods, it has been shown that distance to the decision boundary of a classifier trained to discriminate neural activation patterns for animate and inanimate objects correlates with observer RTs for the same animacy categorization task [Ritchie, J. B., Tovar, D. A., & Carlson, T. A. Emerging object representations in the visual system predict reaction times for categorization. PLoS Computational Biology, 11, e1004316, 2015; Carlson, T. A., Ritchie, J. B., Kriegeskorte, N., Durvasula, S., & Ma, J. Reaction time for object categorization is predicted by representational distance. Journal of Cognitive Neuroscience, 26, 132-142, 2014]. Using MEG decoding, we tested if the same relationship holds when a stimulus manipulation (degradation) increases task difficulty, which we predicted would systematically decrease the distance of activation patterns from the decision boundary and increase RTs. In addition, we tested whether distance to the classifier boundary correlates with drift rates in the linear ballistic accumulator [Brown, S. D., & Heathcote, A. The simplest complete model of choice response time: Linear ballistic accumulation. Cognitive Psychology, 57, 153-178, 2008]. We found that distance to the classifier boundary correlated with RT, accuracy, and drift rates in an animacy categorization task. Split by animacy, the correlations between brain and behavior were sustained longer over the time course for animate than for inanimate stimuli. Interestingly, when examining the distance to the classifier boundary during the peak correlation between brain and behavior, we found that only degraded versions of animate, but not inanimate, objects had systematically shifted toward the classifier decision boundary as predicted. Our results support an asymmetry in the representation of animate and inanimate object categories in the human brain.

A Two-Tier Classification Model for Financial Fraud Detection

Financial fraud has become a daunting challenge for the business companies and baking organizations worldwide. The development of new technologies has provided further and more complicated ways in which criminals commit fraud that result in disastrous consequences. In this paper, we propose a Linear Discriminant Analysis-based novel financial fraud detection model which performs a two-tier classification based on three separate linear discriminant functions. Each function performs its own classification based on the training data and derives its own decision boundary for classification. Then, our two-tier model takes the final classification decision by utilizing the individual decisions of these discriminant functions. We evaluate the performance of our model using reallife datasets in terms of several standard metrics. Besides, we compare the performance of our model with that of several other models found in the literature. Our experimental results suggest that our model achieve reasonably improved classification performance compared to the state-of-the-art ones.

Analysis and convergence theorem of complex quadratic form as decision boundary for data classification

The application of the complex quadratic form as the decision boundary for complex-valued data classification is described. This function is always real when its matrix is Hermitian. Thus, a simple sign function to classify the input data is used. This matrix is obtained through an iterative learning process similar to the Rosenblatt algorithm. The concept of the Frobenius matrix norm is used to prove that the proposed learning algorithm converges if a solution exists. This approach is different from other complex-valued neural networks that use optimisation techniques or feature mapping. An artificial neuron that uses a complex quadratic form as the decision boundary is called a complex quadratic neural unit.

Using Discriminative Dimensionality Reduction to Visualize Classifiers

Albeit automated classifiers offer a standard tool in many application areas, there exists hardly a generic possibility to directly inspect their behavior, which goes beyond the mere classification of (sets of) data points. In this contribution, we propose a general framework how to visualize a given classifier and its behavior as concerns a given data set in two dimensions. More specifically, we use modern nonlinear dimensionality reduction (DR) techniques to project a given set of data points and their relation to the classification decision boundaries. Furthermore, since data are usually intrinsically more than two-dimensional and hence cannot be projected to two dimensions without information loss, we propose to use discriminative DR methods which shape the projection according to given class labeling as is the case for a classification setting. With a given data set, this framework can be used to visualize any trained classifier which provides a probability or certainty of the classification together with the predicted class label. We demonstrate the suitability of the framework in the context of different dimensionality reduction techniques, in the context of different attention foci as concerns the visualization, and as concerns different classifiers which should be visualized.

A fast prototype reduction method based on template reduction and visualization-induced self-organizing map for nearest neighbor algorithm

The k nearest neighbor is a lazy learning algorithm that is inefficient in the classification phase because it needs to compare the query sample with all training samples. A template reduction method is recently proposed that uses only samples near the decision boundary for classification and removes those far from the decision boundary. However, when class distributions overlap, more border samples are retrained and it leads to inefficient performance in the classification phase. Because the number of reduced samples are limited, using an appropriate feature reduction method seems a logical choice to improve classification time. This paper proposes a new prototype reduction method for the k nearest neighbor algorithm, and it is based on template reduction and ViSOM. The potential property of ViSOM is displaying the topology of data on a two-dimensional feature map, it provides an intuitive way for users to observe and analyze data. An efficient classification framework is then presented, which combines the feature reduction method and the prototype selection algorithm. It needs a very small data size for classification while keeping recognition rate. In the experiments, both of synthetic and real datasets are used to evaluate the performance. Experimental results demonstrate that the proposed method obtains above 70 % speedup ratio and 90 % compression ratio while maintaining similar performance to kNN.

Query strategies for evading convex-inducing classifiers

Classifiers are often used to detect miscreant activities. We study how an adversary can systematically query a classifier to elicit information that allows the attacker to evade detection while incurring a near-minimal cost of modifying their intended malfeasance. We generalize the theory of Lowd and Meek (2005) to the family of convex-inducing classifiers that partition their feature space into two sets, one of which is convex. We present query algorithms for this family that construct undetected instances of approximately minimal cost using only polynomially-many queries in the dimension of the space and in the level of approximation. Our results demonstrate that nearoptimal evasion can be accomplished for this family without reverse engineering the classifier's decision boundary. We also consider general lp costs and show that near-optimal evasion on the family of convex-inducing classifiers is generally efficient for both positive and negative convexity for all levels of approximation if p = 1.

Pattern selection approaches for the logical analysis of data considering the outliers and the coverage of a pattern

The logical analysis of data (LAD) is one of the most promising data mining methods developed to date for extracting knowledge from data. The key feature of the LAD is the capability of detecting hidden patterns in the data. Because patterns are basically combinations of certain attributes, they can be used to build a decision boundary for classification in the LAD by providing important information to distinguish observations in one class from those in the other. The use of patterns may result in a more stable performance in terms of being able to classify both positive and negative classes due to their robustness to measurement errors. The LAD technique, however, tends to choose too many patterns by solving a set covering problem to build a classifier; this is especially the case when outliers exist in the data set. In the set covering problem of the LAD, each observation should be covered by at least one pattern, even though the observation is an outlier. Thus, existing approaches tend to select too many patterns to cover these outliers, resulting in the problem of overfitting. Here, we propose new pattern selection approaches for LAD that take both outliers and the coverage of a pattern into account. The proposed approaches can avoid the problem of overfitting by building a sparse classifier. The performances of the proposed pattern selection approaches are compared with existing LAD approaches using several public data sets. The computational results show that the sparse classifiers built on the patterns selected by the proposed new approaches yield an improved classification performance compared to the existing approaches, especially when outliers exist in the data set.

Comparisons of classification methods in the original and pattern spaces

The logical analysis of data (LAD) is one of the most promising data mining and machine learning techniques developed to date for extracting knowledge from data. The LAD is based on the concepts of combinatorics, optimization, and Boolean functions. The key feature of the LAD is the capability of detecting hidden patterns in the data. Since patterns are basically combinations of certain attributes, they can be used to build a decision boundary for classification in the LAD by providing important information to distinguish observations in one class from those in the other class. The use of patterns may result in a more stable performance in terms of being able to classify both positive and negative classes due to their robustness to measurement errors. The patterns are also interpretable and can serve as an essential tool for understanding the problem. These desirable properties of the patterns generated from the LAD motivate the use of the LAD patterns as input variables to other classification techniques to achieve a more stable and accurate performance. In this paper, the patterns generated from the LAD are used as the input variables to the decision tree and k-nearest neighbor classification methods. The applicability and usefulness of the LAD patterns for classification are investigated experimentally. The classification accuracy and sensitivity of the classification results for different classifiers in the original and pattern spaces are compared using several public data. The experimental results show that classifications in the pattern space can yield better and stable performance than those in the original space in terms of accuracy when the classification accuracy of the LAD is relatively good (i.e., the LAD patterns are of good quality), the ratio of the number of patterns to the total number of attributes is small, or the data set for classification is balanced between two classes.