Multi-class Data Research Articles

Manifold regularization based on Nyström type subsampling

In this paper, we study the Nystr{\"o}m type subsampling for large scale kernel methods to reduce the computational complexities of big data. We discuss the multi-penalty regularization scheme based on Nystr{\"o}m type subsampling which is motivated from well-studied manifold regularization schemes. We develop a theoretical analysis of multi-penalty least-square regularization scheme under the general source condition in vector-valued function setting, therefore the results can also be applied to multi-task learning problems. We achieve the optimal minimax convergence rates of multi-penalty regularization using the concept of effective dimension for the appropriate subsampling size. We discuss an aggregation approach based on linear function strategy to combine various Nystr{\"o}m approximants. Finally, we demonstrate the performance of multi-penalty regularization based on Nystr{\"o}m type subsampling on Caltech-101 data set for multi-class image classification and NSL-KDD benchmark data set for intrusion detection problem.

A QoS Based Adaptive Backoff Scheme for Vehicular Ad Hoc Networks.

The media access control (MAC) protocol is a key element in the design of vehicular ad hoc networks (VANETs) that directly affects the network performance. The backoff schemes of existing MAC protocols apply the single backoff process and therefore are not suitable for multi-class data transmission. Additionally, they cannot satisfy the delay requirements of emergency data in the case of varying number of vehicles, causing an adverse effect to the intelligent transportation system (ITS). This paper presents a priority-based adaptive backoff scheme that can enhance the binary exponential backoff (BEB) algorithm as well as the polynomial backoff (QB) algorithm. This system distinguishes priority data with different delay requirements first and designs different backoff schemes for each type of data later. The two-dimensional Markov Chain is used to analyze the backoff scheme and determine the expressions for throughput and delay. The simulation results show that the backoff scheme provided by this paper can reduce the average data delay and regulate each kind of data delay adaptively, according to the varying number of vehicles and different delay requirements.

A sequential three-way approach to multi-class decision

Based on the notion of information and knowledge granularity, sequential three-way decisions can be interpreted and implemented as a multilevel trisecting-and-acting framework which leads to a faster decision-making with the less cost of decision process. At each level, objects are divided into the triplet, consisting of three pair-wise disjoint regions. By considering different combinations of regions, the uncertain objects should be further investigated at the next level due to insufficient information. In fact, sequential strategy is an effective idea to deal with multi-class problem in granular computing. On the basis of this recognition, we focus on a sequential three-way approach to multi-class decision in this paper. To solve the issues of unequal cost and rule conflict simultaneously, we present a novel matrix-based multi-class decision method based on Bayesian decision procedure. Subsequently, a sequence of the proposed multi-class models is conducted under a multilevel granular structure, and objects are assigned into one certain decision class gradually from the lower levels to the higher levels via a top-down manner. Finally, the results of empirical study involving six multi-class data sets on UCI are reported to validate the feasibility and effectiveness of our proposal.

Multiobjective Support Vector Machines: Handling Class Imbalance With Pareto Optimality.

Support vector machines (SVMs) seek to optimize three distinct objectives: maximization of margin, minimization of regularization from the positive class, and minimization of regularization from the negative class. The right choice of weightage for each of these objectives is critical to the quality of the classifier learned, especially in case of the class imbalanced data sets. Therefore, costly parameter tuning has to be undertaken to find a set of suitable relative weights. In this brief, we propose to train SVMs, on two-class as well as multiclass data sets, in a multiobjective optimization framework called radial boundary intersection to overcome this shortcoming. The experimental results suggest that the radial boundary intersection-based scheme is indeed effective in finding the best tradeoff among the objectives compared with parameter-tuning schemes.

Deep Weighted Extreme Learning Machine

The imbalanced data classification attracts increasing attention in the past years due to the continuous expansion of data available in many areas, such as biomedical engineering, surveillance, and computer vision. Learning from imbalanced data is challenging as most standard algorithms fail to properly represent the inherent complex characteristics of the data distribution. As an emerging technology, the extreme learning machine (ELM) and its variants, including the weighted ELM (WELM) and the boosting weighted ELM (BWELM), have been recently developed for the classification of imbalanced data. However, the WELM suffers the following deficiencies: (i) the sample weight matrix is manually chosen and fixed during the learning phase; (ii) the representation capability, namely the capability to extract features or useful information from the original data, is insufficiently explored due to its shallow structure. The BWELM employs the AdaBoost algorithm to optimize the sample weights. But the representation capability is still restricted by the shallow structure. To alleviate these deficiencies, we propose a novel deep weighted ELM (DWELM) algorithm for imbalanced data classification in this paper. An enhanced stacked multilayer deep representation network trained with the ELM (EH-DrELM) is first proposed to improve the representation capability, and a fast AdaBoost algorithm for imbalanced multiclass data (AdaBoost-ID) is developed to optimize the sample weights. Then, the novel DWELM for the imbalance learning is obtained by combining the above two algorithms. Experimental results on nine imbalanced binary-class datasets, nine imbalanced multiclass datasets, and five large benchmark datasets (three for multiclass and two for binary-class) show that the proposed DWELM achieves a better performance than the WELM and BWELM, as well as several state-of-the-art multilayer network-based learning algorithms.

LRID: A new metric of multi-class imbalance degree based on likelihood-ratio test

In this paper, we introduce a new likelihood ratio imbalance degree (LRID) to measure the class-imbalance extent of multi-class data. Imbalance ratio (IR) is usually used to measure class-imbalance extent in imbalanced learning problems. However, IR cannot capture the detailed information in the class distribution of multi-class data, because it only utilises the information of the largest majority class and the smallest minority class. Imbalance degree (ID) has been proposed to solve the problem of IR for multi-class data. However, we note that improper use of distance metric in ID can have harmful effect on the results. In addition, ID assumes that data with more minority classes are more imbalanced than data with less minority classes, which is not always true in practice. Thus ID cannot provide reliable measurement when the assumption is violated. In this paper, we propose a new metric based on the likelihood-ratio test, LRID, to provide a more reliable measurement of class-imbalance extent for multi-class data. Experiments on both simulated and real data show that LRID is competitive with IR and ID, and can reduce the negative correlation with F1 scores by up to 0.55.

Multi-class K-support Vector Nearest Neighbor for Mango Leaf Classification

K-Support Vector Nearest Neighbor (K-SVNN) is one of methods for training data reduction that works only for binary class. This method uses Left Value (LV) and Right Value (RV) to calculate Significant Degree (SD) property. This research aims to modify the K-SVNN for multi-class training data reduction problem by using entropy for calculating SD property. Entropy can measure the impurity of data class distribution, so the selection of the SD can be conducted based on the high entropy. In order to measure performance of the modified K-SVNN in mango leaf classification, experiment is conducted by using multi-class Support Vector Machine (SVM) method on training data with and without reduction. The experiment is performed on 300 mango leaf images, each image represented by 260 features consisting of 256 Weighted Rotation- and Scale-invariant Local Binary Pattern features with average weights (WRSI-LBP-avg) texture features, 2 color features, and 2 shape features. The experiment results show that the highest accuracy for data with and without reduction are 71.33% and 71.00% respectively. It is concluded that K-SVNN can be used to reduce data in multi-class classification problem while preserve the accuracy. In addition, performance of the modified K-SVNN is also compared with two other methods of multi-class data reduction, i.e. Condensed Nearest Neighbor Rule (CNN) and Template Reduction KNN (TRKNN). The performance comparison shows that the modified K-SVNN achieves better accuracy.

Fuzzy Clustering Methods for Categorical Multivariate Data Based on q-Divergence

This paper presents two fuzzy clustering algorithms for categorical multivariate data based on q-divergence. First, this study shows that a conventional method for vectorial data can be explained as regularizing another conventional method using q-divergence. Second, based on the known results that Kullback-Leibler (KL)-divergence is generalized into the q-divergence, and two conventional fuzzy clustering methods for categorical multivariate data adopt KL-divergence, two fuzzy clustering algorithms for categorical multivariate data that are based on q-divergence are derived from two optimization problems built by extending the KL-divergence in these conventional methods to the q-divergence. Through numerical experiments using real datasets, the proposed methods outperform the conventional methods in term of clustering accuracy.

Multiclass Nonnegative Matrix Factorization for Comprehensive Feature Pattern Discovery.

In this big data era, interpretable machine learning models are strongly demanded for the comprehensive analytics of large-scale multiclass data. Characterizing all features from such data is a key but challenging step to understand the complexity. However, existing feature selection methods do not meet this need. In this paper, to address this problem, we propose a Bayesian multiclass nonnegative matrix factorization (MC-NMF) model with structured sparsity that is able to discover ubiquitous and class-specific features. Variational update rules were derived for efficient decomposition. In order to relieve the need of model selection and stably describe feature patterns, we further propose MC-NMF with stability selection, an ensemble method that collectively detects feature patterns from many runs of MC-NMF using different hyperparameter values and training subsets. We assessed our models on both simulated count data and multitumor ribonucleic acid-seq data. The experiments revealed that our models were able to recover predefined feature patterns from the simulated data and identify biologically meaningful patterns from the pan-cancer data.

P41 - I Really Should Change My Mind About These Vegetables: A Signature Assignment with Social Responsibility in Mind

Objective: Describe an exercise in an undergraduate nutrition class that meets Texas Higher Education Coordinating Board (THECB) social responsibility objectives and is assessed using a rubric adapted from the Association of American Colleges and Universities (AACU) VALUES rubrics. Target Audience: The audience is educators interested in embedded assignments and AACU VALUES rubrics for assessing written student work. Theory, Prior Research, Rationale: Even as Every Student Succeeds Act [ESSA] 2015 addressed standardized testing as an accountability measure, it sparked interest in authentic or direct evaluation of student performance in higher education. Through signature assignments universities, engage faculty in assessment design and generate common data across diverse courses and multiple disciplines. Description of Course and Curriculum: Principles of Nutrition is a large enrollment undergraduate course that meets THECB social responsibility objectives including intercultural competence, knowledge of civic responsibility, and ability to engage effectively in regional, national, and global communities. During summer 2017, faculty attended a LEAP Texas Signature Assignment Institute and worked across disciplines to create student learning experiences that demonstrated knowledge acquisition and real-world application. The goal was to produce assignments that tested proficiency without relying on standardized testing. The resulting nutrition signature assignment, I Really Should Change My Mind About These Vegetables, was launched in Fall 2017. Evaluation: Prior to writing a 500-word Change My Mind About These Vegetables reasoned statement, students engaged in Least Likely Vegetable discussions that demonstrated growth in food knowledge and self-reflection concerning the personal and social value of vegetables in the diet. Subsequently a university social responsibility rubric was used to score statements where student described the influence that sharing vegetable likes and dislikes had on personal rules and biases. Using a standard rubric allowed for both assessment at the class level and aggregation of multi-class performance data at the university level. Conclusions and Implications: Signature assignments enhance teaching and learning and contribute to university analytic studies on student success in achieving identified learning outcomes. Funding: None.

Data Space Adaptation for Multiclass Motor Imagery-based BCI.

Various adaptation techniques have been proposed to address the non-stationarity issue faced by electroencephalogram (EEG)-based brain-computer interfaces (BCIs). However, most of these adaptation techniques are only suitable for binary-class BCIs. This paper proposes a supervised multiclass data space adaptation technique (MDSA) to transform the test data using a linear transformation such that the distribution difference between the multiclass train and test data is minimized. The results of using the proposed MDSA on BCI Competition IV dataset 2a improved the classification accuracy by an average of 4.3% when 20 trials per class were used from the test session to estimate adaptation transformation. The results also showed that the proposed MDSA algorithm outperformed the multi pooled mean linear discrimination (MPMLDA) technique with as few as 10 trials per class used for calculating the transformation matrix. Hence the results showed the effectiveness of the proposed MDSA algorithm in addressing non-stationarity issue for multiclass EEG-based BCI.

An empirical comparison on state-of-the-art multi-class imbalance learning algorithms and a new diversified ensemble learning scheme

Class-imbalance learning is one of the most challenging problems in machine learning. As a new and important direction in this field, multi-class imbalanced data classification has attracted a great many research focus in recent years. In this paper, we first make a very comprehensive review on state-of-the-art classification algorithms for multi-class imbalanced data. Moreover, we propose a new multi-class imbalance classification algorithm, which is hereafter referred to as the Diversified Error Correcting Output Codes (DECOC) method. The main idea of DECOC is to combine the improved ECOC (Error Correcting Output Codes) method for tackling class imbalance, and the diversified ensemble learning framework, which finds the best classification algorithm (out of many heterogeneous classification algorithms) for each individual sub-dataset resampled from the original data. We conduct experiments on 19 public datasets to empirically compare the performance of DECOC with 17 state-of-the-art multi-class imbalance learning algorithms, using 4 different accuracy measures: overall accuracy, Geometric mean, F-measure, and Area Under Curve. Experimental results demonstrate that DECOC achieves significantly better accuracy performance than the other 17 algorithms on these accuracy metrics. To advance research in this field, we make all the source codes of DECOC and the above-mentioned 17 state-of-the-art algorithms for imbalanced data classification be available at GitHub: https://github.com/chongshengzhang/Multi_Imbalance.

Bayesian inference on group differences in multivariate categorical data

Multivariate categorical data are common in many fields. An illustrative example is provided by election polls studies assessing evidence of changes in voters’ opinions with their candidates preferences in the 2016 United States Presidential primaries or caucuses. Similar goals arise in routine applications, but current literature lacks a general methodology which combines flexibility, efficiency, and tractability in testing for group differences in multivariate categorical data at different – potentially complex – scales. This contribution addresses such goal by leveraging a Bayesian representation, which factorizes the joint probability mass function for the group variable and the multivariate categorical data as the product of the marginal probabilities for the groups and the conditional probability mass function of the multivariate categorical data, given the group membership. To enhance flexibility, the conditional probability mass function of the multivariate categorical data is defined via a group-dependent mixture of tensor factorizations which facilitates dimensionality reduction and borrowing of information, while providing tractable procedures for computation, and accurate tests assessing global and local group differences. The proposed methods are compared with popular competitors, and the improved performance is outlined in simulations and in American election polls studies.

Postboosting Using Extended G-Mean for Online Sequential Multiclass Imbalance Learning.

In this paper, a novel learning method called postboosting using extended G-mean (PBG) is proposed for online sequential multiclass imbalance learning (OS-MIL) in neural networks. PBG is effective due to three reasons. 1) Through postadjusting a classification boundary under extended G-mean, the challenging issue of imbalanced class distribution for sequentially arriving multiclass data can be effectively resolved. 2) A newly derived update rule for online sequential learning is proposed, which produces a high G-mean for current model and simultaneously possesses almost the same information of its previous models. 3) A dynamic adjustment mechanism provided by extended G-mean is valid to deal with the unresolved challenging dense-majority problem and two dynamic changing issues, namely, dynamic changing data scarcity (DCDS) and dynamic changing data diversity (DCDD). Compared to other OS-MIL methods, PBG is highly effective on resolving DCDS, while PBG is the only method to resolve dense-majority and DCDD. Furthermore, PBG can directly and effectively handle unscaled data stream. Experiments have been conducted for PBG and two popular OS-MIL methods for neural networks under massive binary and multiclass data sets. Through the analyses of experimental results, PBG is shown to outperform the other compared methods on all data sets in various aspects including the issues of data scarcity, dense-majority, DCDS, DCDD, and unscaled data.

Efficient DANNLO classifier for multi-class imbalanced data on Hadoop

In recent years, multi-class imbalance data classification is a major problem in big data. In such situations, we focused on developing a new Deep Artificial Neural Network Learning Optimization (DANNLO) Classifier for large collection of imbalanced data. In our proposed work, first the dataset reduction using principal component analysis for dimensionality reduction and initial centroid is computed. Then, parallel hierarchical pillar k-means clustering algorithm based on MapReduce is used to partitioning of an imbalanced data set into similar subset, which can improve the computational cost. The resultant clusters are given as input to the deep ANN for learning. In the next stage, deep neural network has been trained using the back propagation algorithm. In order to optimize the n-dimensional weight space, firefly optimization algorithm is used. Attractiveness and distance of each firefly is computed. Hadoop is used to handle these large volumes of variable size data. Imbalanced datasets is taken from ECDC (European Centre for Disease Prevention and Control) repository. The experimental results illustrated that the proposed method can significantly improve the effectiveness in classifying imbalanced data based on TP rate, F-measure, G-mean measures, confusion matrix, precision, recall, and ROC. The experimental results suggests that DANNLO classifier exceed other ordinary classifiers such as SVM and Random forest classifier on tested imbalanced data sets.

An exploded view paradigm to disambiguate scatterplots

Small multiples is a popular visualization technique for dealing with overdraw in multi-class data. Small multiples are great at showing pieces of data individually, however, they do not explain how the different pieces fit together. They can also be difficult to understand for unacquainted users. We propose an interactive technique which uses the paradigm of exploded views to make small multiples visualizations more intelligible for unacquainted users. An exploded view is a drawing in which the different components of the object are separated by distance in such a way that the relationship between these components becomes apparent and hidden components of the data are revealed. We use the exploded view paradigm to create various animation designs for multi-class data. The designs are then compared using the Elo ranking scheme. We hypothesize that the exploded view animations increase the ability of users to appreciate the relations among data clusters (in the compound view) and at the same time get a clearer idea about the features of the individual data clusters (in the exploded view). We conduct a user study to compare this interactive approach with a compound view and an animated small multiples visualization.

Dirichlet Process Mixture Models for Modeling and Generating Synthetic Versions of Nested Categorical Data

We present a Bayesian model for estimating the joint distribution of multivariate categorical data when units are nested within groups. Such data arise frequently in social science settings, for example, people living in households. The model assumes that (i) each group is a member of a group-level latent class, and (ii) each unit is a member of a unit-level latent class nested within its group-level latent class. This structure allows the model to capture dependence among units in the same group. It also facilitates simultaneous modeling of variables at both group and unit levels. We develop a version of the model that assigns zero probability to groups and units with physically impossible combinations of variables. We apply the model to estimate multivariate relationships in a subset of the American Community Survey. Using the estimated model, we generate synthetic household data that could be disseminated as redacted public use files. Supplementary materials (Hu et al., 2017) for this article are available online.

Bayesian Non-Parametric Generation Of Fully Synthetic Multivariate Categorical Data in the Presence of Structural Zeros

SummaryStatistical agencies are increasingly adopting synthetic data methods for disseminating microdata without compromising the privacy of respondents. Crucial to the implementation of these approaches are flexible models, able to capture the nuances of the multivariate structure in the original data. In the case of multivariate categorical data, preserving this multivariate structure also often involves satisfying constraints in the form of combinations of responses that cannot logically be present in any data set—like married toddlers or pregnant men—also known as structural zeros. Ignoring structural zeros can result in both logically inconsistent synthetic data and biased estimates. Here we propose the use of a Bayesian non-parametric method for generating discrete multivariate synthetic data subject to structural zeros. This method can preserve complex multivariate relationships between variables, can be applied to high dimensional data sets with massive collections of structural zeros, requires minimal tuning from the user and is computationally efficient. We demonstrate our approach by synthesizing an extract of 17 variables from the 2000 US census. Our method produces synthetic samples with high analytic utility and low disclosure risk.

Problem formulations and solvers in linear SVM: a review

Support vector machine (SVM) is an optimal margin based classification technique in machine learning. SVM is a binary linear classifier which has been extended to non-linear data using Kernels and multi-class data using various techniques like one-versus-one, one-versus-rest, Crammer Singer SVM, Weston Watkins SVM and directed acyclic graph SVM (DAGSVM) etc. SVM with a linear Kernel is called linear SVM and one with a non-linear Kernel is called non-linear SVM. Linear SVM is an efficient technique for high dimensional data applications like document classification, word-sense disambiguation, drug design etc. because under such data applications, test accuracy of linear SVM is closer to non-linear SVM while its training is much faster than non-linear SVM. SVM is continuously evolving since its inception and researchers have proposed many problem formulations, solvers and strategies for solving SVM. Moreover, due to advancements in the technology, data has taken the form of ‘Big Data’ which have posed a challenge for Machine Learning to train a classifier on this large-scale data. In this paper, we have presented a review on evolution of linear support vector machine classification, its solvers, strategies to improve solvers, experimental results, current challenges and research directions.

Classification of Multi-Class BCI Data by Common Spatial Pattern and Fuzzy System

Improving classification accuracy of motor imagery-based brain computer interface (MI-BCI) systems has been discussed widely in the BCI research community. Analyses of multi-class MI data are challenging because feature extraction and classification of these data are more difficult as compared with those applied to binary-class data. This paper introduces a new model for multi-class MI-BCI data classification. The approach encompasses two main stages: feature extraction and fuzzy classification. In the feature extraction phase, a common spatial pattern algorithm is employed to extract significant discriminant features from the multi-class data. The obtained features serve as inputs to a fuzzy classifier that is designed by fusing the fuzzy standard additive model and particle swarm optimization method. The proposed fuzzy model aims to handle uncertainty, noise, and outliers existed in MI-BCI data. Experimental studies are carried out using two benchmark BCI competition data sets. Our results show the effectiveness of the fuzzy classifier against its competing techniques. The proposed model paths the way for paralysed or disordered patients to use movement imagery to control different assistive devices in their daily activities.