Complex Decision Boundaries Research Articles

Overlap Aware Compressed Signal Classification

Compressed signal processing (CSP) is a branch of compressive sensing (CS), which gives a direction to solve a class of signal processing problems directly from the compressive measurements of a signal. CSP utilizes the information preserved in the compressive measurements of a signal to solve certain inference problems like: classification, detection, and estimation, without reconstructing the original signal. It further simplifies the signal processing compared to conventional CS by omitting their complex reconstruction stage. This, in turn, reduces the implementation complexity of signal processing systems. This paper investigates the performance of CSP for classification application. After extracting the features from compressive measurements, these features or the data instances are used for classification purpose. Through experimental analysis, it has been found that as the CS undersampling factor is increased, the overlapping among the data instances predominates. This results in a complex decision boundary, which in turn degrades the classification accuracy at higher undersampling factors. To overcome the above issue, this paper proposes the use of a machine learning method known as overlap aware learning along with CSP. This generates a smoother decision boundary and hence improves the classification accuracy at higher undersampling factors. The simulation results show the trend of improved classification accuracy using the proposed method. An analysis of the proposed method has been done on different datasets and based on run-time complexity and complexity vs gain analysis to verify the effectiveness of proposed method.

A Data-Driven Deep Learning Model to Pattern Recognition for Medical Diagnosis, by using Model Aggregation and Model Selection

In recent years, pattern recognition becomes a research area with increasing importance using several techniques. One of the most common techniques used is Deep Learning. This paper presents a new Deep Learning model to pattern recognition for medical diagnosis. The uncovering of hidden structures is performed by feature selection, model aggregation, and model selection. Our Deep Learning model has the ability to reach the optimal solution and create complex decision boundaries when used to look for and diagnose breast cancer. The evaluation, based on 10-fold cross-validation, showed that the proposed model, which is named BaggingSMF, yielded good results and performed better than Radial Basis Function, Bidirectional Associative Memory, and ELMAN Neural Networks. Experimental studies demonstrate the multidisciplinary applications of our model.

Similarity-based decision tree induction method and its application to cancer recognition on tomographic images

The paper proposes a pattern recognition method using a modification of the class of logical decision functions presented in the form of decision tree. Instead of standard statements corresponding to the tree nodes, in which a variable is tested for a certain set of its values, a more general type of statements is used regarding the similarity of the point in question to different subsets of the observations. At the same time, to determine the degree of similarity, various metrics and subspaces of features can be used. This type of decision tree allows one to obtain more complex decision boundaries, which at the same time have a clear logical interpretation for the user. Several tree induction strategies are considered based on data transformation using support points selected with Relief, SVM, and k -means procedures. The method is experimentally investigated on the problem of tomographic images analysis, as well as on several synthetic datasets. Experiments have shown that the proposed method gives more accurate predictions than CART, SVM, kNN classifiers and deep convolutional neural network (AlexNet).

A research on classification performance of fuzzy classifiers based on fuzzy set theory

Due to the complexities of objects and the vagueness of the human mind, it has attracted considerable attention from researchers studying fuzzy classification algorithms. In this paper, we propose a concept of fuzzy relative entropy to measure the divergence between two fuzzy sets. Applying fuzzy relative entropy, we prove the conclusion that patterns with high fuzziness are close to the classification boundary. Thus, it plays a great role in classification problems that patterns with high fuzziness are classified correctly. Meanwhile, we draw a conclusion that the fuzziness of a pattern and the uncertainty of its class label are equivalent. As is well known, entropy not only measures the uncertainty of random variable, but also represents the amount of information carried by the variable. Hence, a fuzzy classifier with high fuzziness would carry much information about training set. Therefore, in addition to some assessment criteria such as classification accuracy, we could study the classification performance from the perspective of the fuzziness of classifier. In order to try to ensure the objectivity in dealing with unseen patterns, we should make full use of information of the known pattern set and do not make too much subjective assumptions in the process of learning. Consequently, for problems with rather complex decision boundaries especially, under the condition that a certain training accuracy threshold is maintained, we demonstrate that a fuzzy classifier with high fuzziness would have a well generalization performance.

Learning from the machine: interpreting machine learning algorithms for point- and extended-source classification

We investigate star-galaxy classification for astronomical surveys in the context of four methods enabling the interpretation of black-box machine learning systems. The first is outputting and exploring the decision boundaries as given by decision tree based methods, which enables the visualization of the classification categories. Secondly, we investigate how the Mutual Information based Transductive Feature Selection (MINT) algorithm can be used to perform feature pre-selection. If one would like to provide only a small number of input features to a machine learning classification algorithm, feature pre-selection provides a method to determine which of the many possible input properties should be selected. Third is the use of the tree-interpreter package to enable popular decision tree based ensemble methods to be opened, visualized, and understood. This is done by additional analysis of the tree based model, determining not only which features are important to the model, but how important a feature is for a particular classification given its value. Lastly, we use decision boundaries from the model to revise an already existing method of classification, essentially asking the tree based method where decision boundaries are best placed and defining a new classification method. We showcase these techniques by applying them to the problem of star-galaxy separation using data from the Sloan Digital Sky Survey (hereafter SDSS). We use the output of MINT and the ensemble methods to demonstrate how more complex decision boundaries improve star-galaxy classification accuracy over the standard SDSS frames approach (reducing misclassifications by up to $\approx33\%$). We then show how tree-interpreter can be used to explore how relevant each photometric feature is when making a classification on an object by object basis.

A Bayesian Approach to Intervention-Based Clustering

An important task for intelligent healthcare systems is to predict the effect of a new intervention on individuals. This is especially true for medical treatments. For example, consider patients who do not respond well to a new drug or have adversary reactions. Predicting the likelihood of positive or negative response before trying the drug on the patient can potentially save his or her life. We are therefore interested in identifying distinctive subpopulations that respond differently to a given intervention. For this purpose, we have developed a novel technique, Intervention-based Clustering, based on a Bayesian mixture model. Compared to the baseline techniques, the novelty of our approach lies in its ability to model complex decision boundaries by using soft clustering, thus predicting the effect for individuals more accurately. It can also incorporate prior knowledge, making the method useful even for smaller datasets. We demonstrate how our method works by applying it to both simulated and real data. Results of our evaluation show that our model has strong predictive power and is capable of producing high-quality clusters compared to the baseline methods.

Multi-style learning with denoising autoencoders for acoustic modeling in the internet of things (IoT)

We propose a multi-style learning (multi-style training+deep learning) procedure that relies on deep denoising autoencoders (DAEs) to extract and organize the most discriminative information in a training database. Traditionally, multi-style training procedures require either collecting or artificially creating data samples (e.g., by noise injection or data combination) and training a deep neural network (DNN) with all of these different conditions. To expand the applicability of deep learning, the present study instead adopts a DAE to augment the original training set. First, a DAE is utilized to synthesize data that captures useful structure in the input distribution. Next, this synthetic data is combined and mixed within the original training set to exploit the powerful capabilities of DNN classifiers to learn the complex decision boundaries in heterogeneous conditions. By assigning a DAE to synthesize additional examples of representative variations, multi-style learning makes class boundaries less sensitive to corruptions by enforcing back-end DNNs to emphasize on the most discriminative patterns. Moreover, this deep learning technique mitigates the cost and time of data collection and is easy to incorporate into the internet of things (IoT). Results showed these data-mixed DNNs provided consistent performance improvements without even requiring any preprocessing on the test sets.

Multivariate Time-Series Classification Using the Hidden-Unit Logistic Model.

We present a new model for multivariate time-series classification, called the hidden-unit logistic model (HULM), that uses binary stochastic hidden units to model latent structure in the data. The hidden units are connected in a chain structure that models temporal dependencies in the data. Compared with the prior models for time-series classification such as the hidden conditional random field, our model can model very complex decision boundaries, because the number of latent states grows exponentially with the number of hidden units. We demonstrate the strong performance of our model in experiments on a variety of (computer vision) tasks, including handwritten character recognition, speech recognition, facial expression, and action recognition. We also present a state-of-the-art system for facial action unit detection based on the HULM.

Method of weak classifiers fuzzy boosting: Iterative learning of quasi-linear algorithmic composition

Method of fuzzy boosting providing iterative weak classifiers selection and their quasi-linear composition construction is presented. The method is based on the combination of boosting and fuzzy integrating techniques, when at each step of boosting weak classifiers are combined by Choquet fuzzy integral. In the proposed FuzzyBoost algorithm 2-additive fuzzy measures were used, and method for their estimation was proposed. Although detailed theoretical verification of proposed algorithm is still absent, the experimental results, made on simulated data models, demonstrate that in the case of complex decision boundaries FuzzyBoost significantly outperforms AdaBoost.

QRNN: $q$ -Generalized Random Neural Network.

Artificial neural networks (ANNs) are widely used in applications with complex decision boundaries. A large number of activation functions have been proposed in the literature to achieve better representations of the observed data. However, only a few works employ Tsallis statistics, which has successfully been applied to various other fields. This paper presents a random neural network (RNN) with q -Gaussian activation functions [ q -generalized RNN (QRNN)] based on Tsallis statistics. The proposed method employs an additional parameter q (called the entropic index) which reflects the degree of nonextensivity. This approach has the flexibility to model complex decision boundaries of different shapes by varying the entropic index. We conduct numerical experiments to analyze the efficiency of QRNN compared with RNNs and several other classical methods. Statistical tests (Wilcoxon and Friedman) are used to validate our results and show that the QRNN performs significantly better than RNNs with different activation functions. In addition, we find that QRNN outperforms many of the compared classical methods, with the exception of support vector machines, in which case it still exhibits a substantial advantage in terms of implementation simplicity and speed.

Effect of label noise in the complexity of classification problems

Noisy data are common in real-world problems and may have several causes, like inaccuracies, distortions or contamination during data collection, storage and/or transmission. The presence of noise in data can affect the complexity of classification problems, making the discrimination of objects from different classes more difficult, and requiring more complex decision boundaries for data separation. In this paper, we investigate how noise affects the complexity of classification problems, by monitoring the sensitivity of several indices of data complexity in the presence of different label noise levels. To characterize the complexity of a classification dataset, we use geometric, statistical and structural measures extracted from data. The experimental results show that some measures are more sensitive than others to the addition of noise in a dataset. These measures can be used in the development of new preprocessing techniques for noise identification and novel label noise tolerant algorithms. We thereby show preliminary results on a new filter for noise identification, which is based on two of the complexity measures which were more sensitive to the presence of label noise.

Disjunctive normal random forests

We develop a novel supervised learning/classification method, called disjunctive normal random forest (DNRF). A DNRF is an ensemble of randomly trained disjunctive normal decision trees (DNDT). To construct a DNDT, we formulate each decision tree in the random forest as a disjunction of rules, which are conjunctions of Boolean functions. We then approximate this disjunction of conjunctions with a differentiable function and approach the learning process as a risk minimization problem that incorporates the classification error into a single global objective function. The minimization problem is solved using gradient descent. DNRFs are able to learn complex decision boundaries and achieve low generalization error. We present experimental results demonstrating the improved performance of DNDTs and DNRFs over conventional decision trees and random forests. We also show the superior performance of DNRFs over state-of-the-art classification methods on benchmark datasets.

Decision tree ensembles based on kernel features

A classifier ensemble is a set of classifiers whose individual decisions are combined to classify new examples. Classifiers, which can represent complex decision boundaries are accurate. Kernel fun...

A Margin-based Model with a Fast Local Searchnewline for Rule Weighting and Reduction in Fuzzynewline Rule-based Classification Systems

Fuzzy Rule-Based Classification Systems (FRBCS) are highly investigated by researchers due to their noise-stability and interpretability. Unfortunately, generating a rule-base which is sufficiently both accurate and interpretable, is a hard process. Rule weighting is one of the approaches to improve the accuracy of a pre-generated rule-base without modifying the original rules. Most of the proposed methods by now, may over-fit on training data due to generating complex decision boundaries. In this paper, a margin-based optimization model is proposed to improve the performance on unseen data. By this model, fixed-size margins are defined along the decision boundaries and the rule weights are adjusted such that the marginal space would be empty of training instances as much as possible. This model is proposed to support the single-winner reasoning method with a special cost-function to remove undesired effects of noisy instances. The model is proposed to be solved by a fast well-known local search method. With this solving method, a huge amount of irrelevant and redundant rules are removed as a side effect.Two artificial and 16 real world datasets from UCI repository are used to show that the proposed method significantly outperforms other methods with proper choice of the margin size, which is the single parameter of this method.

Ensemble of Nested Dichotomies 기법을 이용한 스마트폰 가속도 센서 데이터 기반의 동작 인지

Ensemble of Nested Dichotomies 기법을 이용한 스마트폰 가속도 센서 데이터 기반의 동작 인지

Reliable All-Pairs Evolving Fuzzy Classifiers

In this paper, we propose a novel design of evolving fuzzy classifiers (EFCs) to handle online multiclass classification problems in a data-streaming context. Therefore, we exploit the concept of all-pairs (AP), a.k.a. all-versus-all, classification using binary classifiers for each pair of classes. This benefits from less complex decision boundaries to be learned, as opposed to a direct multiclass approach, and achieves a higher efficiency in terms of incremental training time than one-versus-rest classification techniques. For the binary classifiers, we apply fuzzy classifiers with singleton class labels in the consequences, as well as Takagi-Sugeno (T-S) fuzzy models to conduct regression on [0, 1] for each class pair. Both are evolved and incrementally trained in a data-streaming context, yielding a permanent update of the whole AP collection of classifiers, thus being able to properly react to dynamic changes in the streams. The classification phase considers a novel strategy by using the preference levels of each pair of classes that are collected in a preference relation matrix and performing a weighted voting scheme on this matrix. This is done by investigating the reliability of the classifiers in their predictions: 1) integrating the degree of ignorance on samples to be classified as weights for the preference levels and 2) new conflict models used in the single binary classifiers and when calculating the final class response based on the preference relation matrix. The advantage of the new EFC concept over the single model (using a direct multiclass classification concept) and multimodel architectures (using a one-versus-rest classification concept) will be underlined by empirical evaluations and comparisons at the end of this paper based on high-dimensional real-world multiclass classification problems. The results also show that integrating conflict and ignorance concepts into the preference relations can boost classifier accuracies.

Predicting treatment efficacy via quantitative magnetic resonance imaging: a Bayesian joint model.

The prognosis for patients with high grade gliomas is poor, with a median survival of 1 year. Treatment efficacy assessment is typically unavailable until 5-6 months post diagnosis. Investigators hypothesize that quantitative magnetic resonance imaging can assess treatment efficacy 3 weeks after therapy starts, thereby allowing salvage treatments to begin earlier. The purpose of this work is to build a predictive model of treatment efficacy by using quantitative magnetic resonance imaging data and to assess its performance. The outcome is 1-year survival status. We propose a joint, two-stage Bayesian model. In stage I, we smooth the image data with a multivariate spatiotemporal pairwise difference prior. We propose four summary statistics that are functionals of posterior parameters from the first-stage model. In stage II, these statistics enter a generalized non-linear model as predictors of survival status. We use the probit link and a multivariate adaptive regression spline basis. Gibbs sampling and reversible jump Markov chain Monte Carlo methods are applied iteratively between the two stages to estimate the posterior distribution. Through both simulation studies and model performance comparisons we find that we can achieve higher overall correct classification rates by accounting for the spatiotemporal correlation in the images and by allowing for a more complex and flexible decision boundary provided by the generalized non-linear model.

Bagged ensemble of Fuzzy C-Means classifiers for nuclear transient identification

This paper presents an ensemble-based scheme for nuclear transient identification. The approach adopted to construct the ensemble of classifiers is bagging; the novelty consists in using supervised fuzzy C-means (FCM) classifiers as base classifiers of the ensemble. The performance of the proposed classification scheme has been verified by comparison with a single supervised, evolutionary-optimized FCM classifier with respect of the task of classifying artificial datasets. The results obtained indicate that in the cases of datasets of large or very small sizes and/or complex decision boundaries, the bagging ensembles can improve classification accuracy. Then, the approach has been applied to the identification of simulated transients in the feedwater system of a boiling water reactor (BWR).

Fuzzy-UCS: A Michigan-Style Learning Fuzzy-Classifier System for Supervised Learning

This paper presents Fuzzy-UCS, a Michigan-style learning fuzzy-classifier system specifically designed for supervised learning tasks. Fuzzy-UCS is inspired by UCS, an on-line accuracy-based learning classifier system. Fuzzy-UCS introduces a linguistic representation of the rules with the aim of evolving more readable rule sets, while maintaining similar performance and generalization capabilities to those presented by UCS. The behavior of Fuzzy-UCS is analyzed in detail from several perspectives. The granularity of the linguistic fuzzy representation to define complex decision boundaries is illustrated graphically, and the test performance obtained with different inference schemes is studied. Fuzzy-UCS is also compared with a large set of other fuzzy and nonfuzzy learners, demonstrating the competitiveness of its on-line architecture in terms of performance and interpretability. Finally, the paper shows the advantages obtained when Fuzzy-UCS is applied to learn fuzzy models from large volumes of data.

Local Classifier Weighting by Quadratic Programming

It has been widely accepted that the classification accuracy can be improved by combining outputs of multiple classifiers. However, how to combine multiple classifiers with various (potentially conflicting) decisions is still an open problem. A rich collection of classifier combination procedures -- many of which are heuristic in nature -- have been developed for this goal. In this brief, we describe a dynamic approach to combine classifiers that have expertise in different regions of the input space. To this end, we use local classifier accuracy estimates to weight classifier outputs. Specifically, we estimate local recognition accuracies of classifiers near a query sample by utilizing its nearest neighbors, and then use these estimates to find the best weights of classifiers to label the query. The problem is formulated as a convex quadratic optimization problem, which returns optimal nonnegative classifier weights with respect to the chosen objective function, and the weights ensure that locally most accurate classifiers are weighted more heavily for labeling the query sample. Experimental results on several data sets indicate that the proposed weighting scheme outperforms other popular classifier combination schemes, particularly on problems with complex decision boundaries. Hence, the results indicate that local classification-accuracy-based combination techniques are well suited for decision making when the classifiers are trained by focusing on different regions of the input space.