Multiclass Logistic Regression Research Articles

Sentiment Analysis for Odd-Even Scheme in Delhi

Objectives: This paper analyzes odd-even traffic scheme using tweets posted on Twitter from December 2015 to August 2016. Twitter is a social network where users post their feelings, opinions and sentiments for any event using hashtags and mentions. The tweets posted publicly can be viewed by anyone interested. This paper transforms the unstructured tweets into structured information using open source libraries. Further objective is to build a model using machine learning classification techniques to classify unseen tweets on the same context. Methods/Analysis: This paper collects tweets on this event under hashtags. This study explores Dandelion Application Programming Interface for annotation of tweets for academic research. This paper uses machine learning classifications approaches for sentiment analysis and opinion mining. This paper presents empirical comparison of three supervised classification algorithms namely, Multinomial Naive Bayes, Support Vector Machines (SVM) and Multiclass Logistic Regression. The performances of these classifiers are evaluated through standard evaluation metrics. Findings: The experimental results reveal that SVM classifier outperforms the other two classification algorithms. This study may help in decision making of this event to some extent. Application: A large number of applications of sentiment and opinion mining can be designed using packages and freely open resources within a time frame now a days.

Sparse group lasso for multiclass functional logistic regression models

ABSTRACTSparsity-inducing penalties are useful tools for variable selection and are also effective for regression problems where the data are functions. We consider the problem of selecting not only variables but also decision boundaries in multiclass logistic regression models for functional data, using sparse regularization. The parameters of the functional logistic regression model are estimated in the framework of the penalized likelihood method with the sparse group lasso-type penalty, and then tuning parameters for the model are selected using the model selection criterion. The effectiveness of the proposed method is investigated through simulation studies and the analysis of a gene expression data set.

Early Destination Prediction with Spatio-temporal User Behavior Patterns

Predicting user behavior makes it possible to provide personalized services. Destination prediction (e.g. predicting a future location) can be applied to various practical applications. An example of destination prediction is personalized GIS services, which are expected to provide alternate routes to enable users to avoid congested roads. However, the destination prediction problem requires critical trade-offs between timing and accuracy. In this paper, we focus on early destination prediction as the central issue, as early recognition in destination prediction has not been fully explored. As an alternative to the traditional two basic approaches with trajectory tracking that narrow down the candidates with respect to the trip progress, and Next Place Prediction (NPP) that infers the future location of a user from user habits, we propose a new probabilistic model based on both conventional models. The advantage of our model is that it drastically narrows down the destination candidates efficiently at the early stage of a trip, owing to the staying information derived from the NPP approach. In other words, our approach achieves high prediction accuracy by considering both approaches at the same time. To implement our model, we employ SubSynE for state-of-the-art prediction based on trajectory tracking as well as a multi-class logistic regression based on user contexts. Despite the simplicity of our model, the proposed method provides improved performance compared to conventional approaches based on the experimental results using the GPS logs of 1,646 actual users from the commercial services.

A Multinomial Logistic Regression Approach for Arrhythmia Detection

Cardiovascular diseases are the leading causes on mortality in the world. Consequently, tools and methods providing useful and applicable insights into their assessment play a crucial role in the prediction and managements of specific heart conditions. In this article, we introduce a method based on multi-class Logistic Regression as a classifier to provide a powerful and accurate insight into cardiac arrhythmia, which is one of the predictors of serious vascular diseases. As suggested by our evaluation, this provides a robust, scalable, and accurate system, which can successfully tackle the challenges posed by the utilisation of big data in the medical sector.

Radiomic analysis reveals DCE-MRI features for prediction of molecular subtypes of breast cancer.

The purpose of this study was to investigate the role of features derived from breast dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) and to incorporated clinical information to predict the molecular subtypes of breast cancer. In particular, 60 breast cancers with the following four molecular subtypes were analyzed: luminal A, luminal B, human epidermal growth factor receptor-2 (HER2)-over-expressing and basal-like. The breast region was segmented and the suspicious tumor was depicted on sequentially scanned MR images from each case. In total, 90 features were obtained, including 88 imaging features related to morphology and texture as well as dynamic features from tumor and background parenchymal enhancement (BPE) and 2 clinical information-based parameters, namely, age and menopausal status. An evolutionary algorithm was used to select an optimal subset of features for classification. Using these features, we trained a multi-class logistic regression classifier that calculated the area under the receiver operating characteristic curve (AUC). The results of a prediction model using 24 selected features showed high overall classification performance, with an AUC value of 0.869. The predictive model discriminated among the luminal A, luminal B, HER2 and basal-like subtypes, with AUC values of 0.867, 0.786, 0.888 and 0.923, respectively. An additional independent dataset with 36 patients was utilized to validate the results. A similar classification analysis of the validation dataset showed an AUC of 0.872 using 15 image features, 10 of which were identical to those from the first cohort. We identified clinical information and 3D imaging features from DCE-MRI as candidate biomarkers for discriminating among four molecular subtypes of breast cancer.

Sparse coding based dense feature representation model for hyperspectral image classification

We present a sparse coding based dense feature representation model (a preliminary version of the paper was presented at the SPIE Remote Sensing Conference, Dresden, Germany, 2013) for hyperspectral image (HSI) classification. The proposed method learns a new representation for each pixel in HSI through the following four steps: sub-band construction, dictionary learning, encoding, and feature selection. The new representation usually has a very high dimensionality requiring a large amount of computational resources. We applied the l 1 /l q regularized multiclass logistic regression technique to reduce the size of the new representation. We integrated the method with a linear support vector machine (SVM) and a composite kernels SVM (CKSVM) to discriminate different types of land cover. We evaluated the proposed algorithm on three well-known HSI datasets and compared our method to four recently developed classification methods: SVM, CKSVM, simultaneous orthogonal matching pursuit, and image fusion and recursive filtering. Experimental results show that the proposed method can achieve better overall and average classification accuracies with a much more compact representation leading to more efficient sparse models for HSI classification.

Semantic Boosting Cross-Modal Hashing for efficient multimedia retrieval

Cross-modal hashing aims to embed data from different modalities into a common low-dimensional Hamming space, which serves as an important part in cross-modal retrieval. Although many linear projection methods were proposed to map cross-modal data into a common abstract space, the semantic similarity between cross-modal data was often ignored. To address this issue, we put forward a novel cross-modal hashing method named Semantic Boosting Cross-Modal Hashing (SBCMH). To preserve the semantic similarity, we first apply multi-class logistic regression to project heterogeneous data into a semantic space, respectively. To further narrow the semantic gap between different modalities, we then use a joint boosting framework to learn hash functions, and finally transform the mapped data representations into a measurable binary subspace. Comparative experiments on two public datasets demonstrate the effectiveness of the proposed SBCMH.

Automatic assessment of voice quality in the context of multiple annotations.

Approaches to evaluate voice quality include perceptual analysis, and acoustical analysis. Perceptual analysis is subjective and depends mostly on the ability of a specialist to assess a pathology, whereas acoustical analysis is objective, but highly relies on the quality of the so called annotations that the specialist assigns to the voice signal. The quality of the annotations for acoustical analysis depends heavily on the expertise and knowledge of the specialist. We face a scenario where we have annotations performed by several specialists with different levels of expertise and knowledge. Traditional pattern recognition methods employed in acoustical analysis are no longer applicable, since these methods are designed for scenarios where a "ground-truth" label is assigned by the specialist. In this paper, we apply recent developments in machine learning for taking into account multiple annotators for acoustical analysis of voice signals. For the classification step we compare two techniques, one of them based on Gaussian Processes for regression with multiple annotators, and the other is a multi-class Logistic Regression model that measures the annotator performance in terms of sensitivity and specificity. The performance of classifiers is assessed in terms of Cohen's Kappa index. Results show that the multi-annotator classification schemes have better performance when compared to techniques based on a traditional classifier where the true label is estimated from the multiple annotations available using majority voting.

New building signature extraction method from single very high-resolution synthetic aperture radar images based on symmetric analysis

To monitor urban areas using a synthetic aperture radar (SAR) sensor, we propose a symmetric analysis-based building signature extraction method. Instead of using separated algorithms, a unified framework is proposed to extract both layover and shadow areas. Since these two primitives usually exhibit long strip patterns in very-high-resolution SAR images, symmetry axes are first delineated. After that, local features are extracted from both symmetry and range direction to better distinguish different primitives. Then, these local radiometric features are used to identify different categories (layover, shadow, and background) via an efficient multiclass logistic regression classifier. To discriminate individual primitives, geometric information is adopted via an improved Ramer Douglas Peucker algorithm, which also simplifies the parameters for describing these primitives. To further enhance accuracy, combinatory analysis is implemented to exclude some false detections, and then shadow areas are extended via a local region growing method. The proposed approach is tested on a 0.75-m resolution airborne C band SAR image. The experiments are carried out under both small- and large-scale scenes, and the comparative results show our method has some advantages in low-contrast target detection and false-alarm elimination.

A novel logistic multi-class supervised classification model based on multi-fractal spectrum parameters for hyperspectral data

A novel logistic multi-class supervised classification model based on multi-fractal spectrum parameters is proposed to avoid the error that is caused by the difference between the real data distribution and the hypothetic Gaussian distribution and avoid the computational burden working in the logistic regression classification directly for hyperspectral data. The multi-fractal spectra and parameters are calculated firstly with training samples along the spectral dimension of hyperspectral data. Secondly, the logistic regression model is employed in our work because the logistic regression classification model is a distribution-free nonlinear model which is based on the conditional probability without the Gaussian distribution assumption of the random variables, and the obtained multi-fractal parameters are applied to establish the multi-class logistic regression classification model. Finally, the Newton–Raphson method is applied to estimate the model parameters via the maximum likelihood algorithm. The classification results of the proposed model are compared with the logistic regression classification model based on an adaptive bands selection method by using the Airborne Visible/Infrared Imaging Spectrometer and airborne Push Hyperspectral Imager data. The results illuminate that the proposed approach achieves better accuracy with lower computational cost simultaneously.

Variable and boundary selection for functional data via multiclass logistic regression modeling

Penalties with an ℓ1 norm provide solutions in which some coefficients are exactly zero and can be used for selecting variables in regression settings. When applied to the logistic regression model, they also can be used to select variables which affect classification. We focus on the form of ℓ1 penalties in logistic regression models for functional data, in particular, their use in classifying functions into three or more groups while simultaneously selecting variables or classification boundaries. We provide penalties that appropriately select the variables in functional multiclass logistic regression models. Analysis of simulation and real data show that the form of the penalty should be selected in accordance with the purpose of the analysis.

Quantitative characterization of protein tertiary motifs

A quantitative feature-vector representation/model of tertiary structural motifs of proteins is presented. Multiclass logistic regression and a probabilistic neural network were employed to apply this representation to large data sets in order to classify them into major families of distinct motif types (including those of functional importance) with high statistical confidence. Scatter plots of random samples of these motifs were obtained through two-dimensional transformation of the feature vector by metric MDS (multidimensional scaling). The plots showed distinct clusters and shapes for different families and demonstrated the relevance and importance of the proposed quantitative feature-vector representation for characterizing protein tertiary structural motifs. The relative importance of the features was analyzed. The scope of the present work to investigate Nature's prioritization and optimization of functional motif structures is highlighted.

Improving cross-modal and multi-modal retrieval combining content and semantics similarities with probabilistic model

With the ongoing development of the internet, a large number of multimedia documents containing images and texts have appeared in the daily life of people. Therefore, how to effectively and efficiently conduct cross-modal and multi-modal retrieval is being an important issue. Although some methods have been proposed to deal with the issue, their retrieval processes are confined to a single information source of multimedia documents, such as the representations of images and texts at a semantic level. In this paper, we propose a novel probabilistic model, namely CCSS, which not only combines low-level content and high-level semantics similarities through a first-order Markov chain, but also provides heterogeneous similarity measures for different unimedia types. The ranked list for a query is obtained by highlighting an optimal path across the chain. Content similarity focuses on the internal structure of each modality, while semantics similarity focuses on the semantic correlation between different modalities. Both of them are significant and their combination can be complementary to each other. Multi-class logistic regression and random forests are used to map the original features of each unimedia into a semantic space. According to the query-by-example scenario, the experiments on the Wikipedia dataset show that the performance of our model significantly outperforms those of state-of-the-art approaches for cross-modal retrieval. Additionally, the proposed multi-modal method is also shown to outperform previous systems on image retrieval task.

Multi-Class Logistic Regression Classifier with BFGS Method

The classical multi-class logistic regression classifier uses Newton method to optimize its loss function and suffers the expensive computations and the un-stable iteration process. In our work, we apply the state-of-art optimization techniques BFGS to train multi-class logistic regression and compare them with Newton method on the classification accuracy of 25 datasets experimentally. The results show that BFGS achieves better classification accuracy than the Newton method. Moreover, BFGS have the lower time complexity, in contrast with Newton method. Finally, we also observe that logistic classifier with BFGS method demonstrate comparable performance with the SVM classifier.

Block coordinate descent algorithms for large-scale sparse multiclass classification

Over the past decade, ℓ 1 regularization has emerged as a powerful way to learn classifiers with implicit feature selection. More recently, mixed-norm (e.g., ℓ 1/ℓ 2) regularization has been utilized as a way to select entire groups of features. In this paper, we propose a novel direct multiclass formulation specifically designed for large-scale and high-dimensional problems such as document classification. Based on a multiclass extension of the squared hinge loss, our formulation employs ℓ 1/ℓ 2 regularization so as to force weights corresponding to the same features to be zero across all classes, resulting in compact and fast-to-evaluate multiclass models. For optimization, we employ two globally-convergent variants of block coordinate descent, one with line search (Tseng and Yun in Math. Program. 117:387–423, 2009) and the other without (Richtárik and Takáč in Math. Program. 1–38, 2012a; Tech. Rep. arXiv:1212.0873 , 2012b). We present the two variants in a unified manner and develop the core components needed to efficiently solve our formulation. The end result is a couple of block coordinate descent algorithms specifically tailored to our multiclass formulation. Experimentally, we show that block coordinate descent performs favorably compared to other solvers such as FOBOS, FISTA and SpaRSA. Furthermore, we show that our formulation obtains very compact multiclass models and outperforms ℓ 1/ℓ 2-regularized multiclass logistic regression in terms of training speed, while achieving comparable test accuracy.

On Optimization of Multi-Class Logistic Regression Classifier

The classical multi-class logistic regression classifier uses Newton method to optimize its loss function and suffers the expensive computations and the un-stable iteration process. In our work, we apply two state-of-art optimization techniques including conjugate gradient (CG) and BFGS to train multi-class logistic regression and compare them with Newton method on the classification accuracy of 20 datasets experimentally. The results show that CG and BFGS achieves better classification accuracy than the Newton method. Moreover, CG and BFGS have the lower time complexity, in contrast with Newton method. Finally, we also observe that CG and BFGS demonstrate similar performance.

Clasificador basado en una máquina de vectores de soporte de mínimos cuadrados frente a un clasificador por regresión logística ante el reconocimiento de dígitos numéricos

En este artículo se compara el desempeño de una máquina de vectores de soporte de mínimos cuadrados multi-clase (multi-class Least Square Support Vector Machine mc-LSSVM) frente a un clasificador por regresión logística multi-clase, ante el problema del reconocimiento de dígitos numéricos (0-9) escritos a mano. Para desarrollar la comparación se usó un set de datos compuesto por 5000 imágenes de dígitos numéricos escritos a mano (500 imágenes por cada número del 0-9), cada imagen de 20 x 20 pixeles. La entrada a cada uno de los sistemas evaluados fueron vectores de dimensión 400, correspondientes a cada imagen (no se realizó extracción de características). Ambos clasificadores utilizan la estrategia Uno contra todos (OneVsAll) para habilitar la multi-clasificación y una función de validación cruzada aleatoria para el proceso de minimización de la función de costo. Las métricas de comparación fueron la precisión y el tiempo de entrenamiento bajo las mismas condiciones computacionales. Ambas técnicas evaluadas presentaron una precisión superior al 95 %, siendo LS-SVM ligeramente más precisa. Sin embargo, en el costo computacional sí se encontró una diferencia notoria: LS-SVM requiere un tiempo de entrenamiento 16,42 % inferior al requerido por el modelo basado en regresión logística bajos las mismas condiciones computacionales.

Mixed-membership naive Bayes models

In recent years, mixture models have found widespread usage in discovering latent cluster structure from data. A popular special case of finite mixture models is the family of naive Bayes (NB) models, where the probability of a feature vector factorizes over the features for any given component of the mixture. Despite their popularity, naive Bayes models do not allow data points to belong to different component clusters with varying degrees, i.e., mixed memberships, which puts a restriction on their modeling ability. In this paper, we propose mixed-membership naive Bayes (MMNB) models. On one hand, MMNB can be viewed as a generalization of NB by putting a Dirichlet prior on top to allow mixed memberships. On the other hand, MMNB can also be viewed as a generalization of latent Dirichlet allocation (LDA) with the ability to handle heterogeneous feature vectors with different types of features, e.g., real, categorical, etc.. We propose two variational inference algorithms to learn MMNB models. The first one is based on ideas originally used in LDA, and the second one uses substantially fewer variational parameters, leading to a significantly faster algorithm. Further, we extend MMNB/LDA to discriminative mixed-membership models for classification by suitably combining MMNB/LDA with multi-class logistic regression. The efficacy of the proposed mixed-membership models is demonstrated by extensive experiments on several datasets, including UCI benchmarks, recommendation systems, and text datasets.

An application of methods for the probabilistic three-class classification of pregnancies of unknown location

We developed and compared classifiers to predict the outcome of pregnancies of unknown location (PUL). This is a three-class problem, as the possible outcomes are failing PUL, intra-uterine pregnancy (IUP), or ectopic pregnancy (EP). We focused on probabilistic classification because of the importance of uncertainty information in medical decision making. Nine methods were implemented, based on logistic regression (LR), multi-layer perceptrons, least squares support vector machines (LS-SVMs), and kernel logistic regression (KLR). The LR models involved manual checks for the necessity of input transformations or interaction effects. The classifiers were developed on the training set (n=508) and evaluated on the test set (n=348). We used two performance measures that only evaluate discriminatory potential, and two that investigate the exact probabilities and/or discriminatory potential. Classifier comparison was carried out using a ranking method. The classifier based on a combination of binary LR models using pairwise coupling (LR-PC) ranked first or second for each performance measure. LR-PC obtained test set areas under the receiver operating characteristic curve of 0.989, 0.988, and 0.924 for the prediction of failing PULs, IUPs, and EPs, respectively. Multi-class LR, multi-class KLR, and a combination of binary Bayesian LS-SVMs with radial basis function kernel were always ranked in the top five. Models with low emphasis on nonlinearity were ranked at the bottom. Importantly, LR-PC also performed better than previously developed classifiers based on multi-class LR. The prediction of PULs was good, most notably for failing PULs and IUPs. Logistic regression models performed remarkably well. Multi-class KLR also performed well, whilst nonlinearity was automatically incorporated and probabilistic outputs were directly given without Bayesian analysis or a combination of binary results. The selected inputs are reasonably objective and easy to obtain in clinical practice. Taken together, this study provided useful decision support tools for implementation in clinical practice.

A general approach to simultaneous model fitting and variable elimination in response models for biological data with many more variables than observations

BackgroundWith the advent of high throughput biotechnology data acquisition platforms such as micro arrays, SNP chips and mass spectrometers, data sets with many more variables than observations are now routinely being collected. Finding relationships between response variables of interest and variables in such data sets is an important problem akin to finding needles in a haystack. Whilst methods for a number of response types have been developed a general approach has been lacking.ResultsThe major contribution of this paper is to present a unified methodology which allows many common (statistical) response models to be fitted to such data sets. The class of models includes virtually any model with a linear predictor in it, for example (but not limited to), multiclass logistic regression (classification), generalised linear models (regression) and survival models. A fast algorithm for finding sparse well fitting models is presented. The ideas are illustrated on real data sets with numbers of variables ranging from thousands to millions. R code implementing the ideas is available for download.ConclusionThe method described in this paper enables existing work on response models when there are less variables than observations to be leveraged to the situation when there are many more variables than observations. It is a powerful approach to finding parsimonious models for such datasets. The method is capable of handling problems with millions of variables and a large variety of response types within the one framework. The method compares favourably to existing methods such as support vector machines and random forests, but has the advantage of not requiring separate variable selection steps. It is also works for data types which these methods were not designed to handle. The method usually produces very sparse models which make biological interpretation simpler and more focused.