Sparse Multinomial Logistic Regression Research Articles

An Introduction to SGTPPR: Sparse Geochemical Tectono‐Magmatic Setting Probabilistic MembershiP DiscriminatoR

AbstractWe present a new and easy‐to‐use geochemical tectono‐magmatic setting discriminator to calculate the probability of membership (the Sparse Geochemical Tectono‐magmatic setting Probabilistic membershiP discriminatoR, SGTPPR) that runs in Excel. It outputs the probability of membership for eight different tectono‐magmatic settings (mid‐ocean ridge, oceanic island, oceanic plateau, continental flood basalt province, intra‐oceanic arc, continental arc, island arc, and back‐arc basin) for a given volcanic rock sample based on major and selected trace element contents (SiO2, TiO2, Al2O3, Fe2O3, MgO, CaO, K2O, Na2O, Rb, Sr, Y, Zr, Nb, and Ba). We consider all possible ratios and multiplications of these contents, in addition to the contents themselves, which improves the discrimination accuracy. We use a statistical method called sparse multinomial logistic regression to construct a robust and predictive discrimination model. By imposing the sparsity, only a small number of essential variables are included in the model. The variables are objectively extracted from 287 possible geochemical variables, including all possible ratios and multiplications of the major and trace element contents. The constructed model exhibits a high classification ability, indicating that tectonic discrimination using major and selected trace elements yields a high classification ability when ratios and multiplications are considered. The system outputs the relative weights of the variables (i.e., contents, and ratios and multiplications of contents) of the input geochemical data to the calculated membership probabilities. This information can be used to evaluate and interpret the results. We apply the model to multiple samples of a geological unit, to determine the tectonic setting.

Raman microspectroscopy and machine learning for use in identifying radiation-induced lung toxicity.

In this work, we explore and develop a method that uses Raman spectroscopy to measure and differentiate radiation induced toxicity in murine lungs with the goal of setting the foundation for a predictive disease model. Analysis of Raman tissue data is achieved through a combination of techniques. We first distinguish between tissue measurements and air pockets in the lung by using group and basis restricted non-negative matrix factorization. We then analyze the tissue spectra using sparse multinomial logistic regression to discriminate between fibrotic gradings. Model validation is achieved by splitting the data into a training set containing 70% of the data and a test set with the remaining 30%; classification accuracy is used as the performance metric. We also explore several other potential classification tasks wherein the response considered is the grade of pneumonitis and fibrosis sickness. A classification accuracy of 91.6% is achieved on the test set of fibrotic gradings, illustrating the ability of Raman measurements to detect differing levels of fibrotic disease among the murine lungs. It is also shown via further modeling that coarser consideration of fibrotic grading via binning (ie. 'Low', 'Medium', 'High') does not degrade performance. Finally, we consider preliminary models for pneumonitis discrimination using the same methodologies.

Classification of tissue biopsies by Raman spectroscopy guided by quantitative phase imaging and its application to bladder cancer.

We present a multimodal label-free optical measurement approach for analyzing sliced tissue biopsies by a unique combination of quantitative phase imaging and localized Raman spectroscopy. First, label-free quantitative phase imaging of the entire unstained tissue slice is performed using automated scanning. Then, pixel-wise segmentation of the tissue layers is performed by a kernelled structural support vector machine based on Haralick texture features, which are extracted from the quantitative phase profile, and used to find the best locations for performing the label-free localized Raman measurements. We use this multimodal label-free measurement approach for segmenting the urothelium in benign and malignant bladder cancer tissues by quantitative phase imaging, followed by location-guided Raman spectroscopy measurements. We then use sparse multinomial logistic regression (SMLR) on the Raman spectroscopy measurements to classify the tissue types, demonstrating that the prior segmentation of the urothelium done by label-free quantitative phase imaging improves the Raman spectra classification accuracy from 85.7% to 94.7%.

PIANO: A fast parallel iterative algorithm for multinomial and sparse multinomial logistic regression

Multinomial Logistic Regression is a well-studied tool for classification and has been widely used in fields like image processing, computer vision and, bioinformatics, to name a few. Under a supervised classification scenario, a Multinomial Logistic Regression model learns a weight vector to differentiate between any two classes by optimizing over the likelihood objective. With the advent of big data, the inundation of data has resulted in large dimensional weight vector and has also given rise to a huge number of classes, which makes the classical methods applicable for model estimation not computationally viable. To handle this issue, we here propose a parallel iterative algorithm: Parallel Iterative Algorithm for MultiNomial LOgistic Regression (PIANO) which is based on the Majorization Minimization procedure, and can parallely update each element of the weight vectors. Further, we also show that PIANO can be easily extended to solve the Sparse Multinomial Logistic Regression problem - an extensively studied problem because of its attractive feature selection property. In particular, we work out the extension of PIANO to solve the Sparse Multinomial Logistic Regression problem with ℓ1 and ℓ0 regularizations. We also prove that PIANO converges to a stationary point of the Multinomial and the Sparse Multinomial Logistic Regression problems. Simulations were conducted to compare PIANO with the existing methods, and it was found that the proposed algorithm performs better than the existing methods in terms of speed of convergence.

Subspace quadratic regularization method for group sparse multinomial logistic regression

Subspace quadratic regularization method for group sparse multinomial logistic regression

Using topic models to identify clients' functioning levels and alliance ruptures in psychotherapy.

Computerized natural language processing techniques can analyze psychotherapy sessions as texts, thus generating information about the therapy process and outcome and supporting the scaling-up of psychotherapy research. We used topic modeling to identify topics discussed in psychotherapy sessions and explored (a) which topics best identified clients' functioning and alliance ruptures and (b) whether changes in these topics were associated with changes in outcome. Transcripts of 873 sessions from 58 clients treated by 52 therapists were analyzed. Before each session, clients self-reported functioning and symptom level. After each session, therapists reported the extent of alliance rupture. Latent Dirichlet allocation was used to extract latent topics from psychotherapy textual data. Then a sparse multinomial logistic regression model was used to predict which topics best identified clients' functioning levels and the occurrence of alliance ruptures in psychotherapy sessions. Finally, we used multilevel growth models to explore the associations between changes in topics and changes in outcome. Session-based processing yielded a list of semantic topics. The model identified the labels above chance (65% to 75% accuracy). Change trajectories in topics were associated with change trajectories in outcome. The results suggest that topic models can exploit rich linguistic data within sessions to identify psychotherapy process and outcomes. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Multiclass Classification by Sparse Multinomial Logistic Regression

In this paper we consider high-dimensional multiclass classification by sparse multinomial logistic regression. We propose first a feature selection procedure based on penalized maximum likelihood with a complexity penalty on the model size and derive the nonasymptotic bounds for misclassification excess risk of the resulting classifier. We establish also their tightness by deriving the corresponding minimax lower bounds. In particular, we show that there is a phase transition between small and large number of classes. The bounds can be reduced under the additional low noise condition. To find a penalized maximum likelihood solution with a complexity penalty requires, however, a combinatorial search over all possible models. To design a feature selection procedure computationally feasible for high-dimensional data, we propose multinomial logistic group Lasso and Slope classifiers and show that they also achieve the minimax order.

Communication‐efficient distributed large‐scale sparse multinomial logistic regression

SummarySparse multinomial logistic regression (SMLR) is widely used in image classification and text classification due to its feature selection and probabilistic output. However, the traditional SMLR algorithm cannot satisfy the memory and time needs of big data, which makes it necessary to propose a new distributed solution algorithm. The existing distributed SMLR algorithm has some shortcomings in network strategy and cannot make full use of the computing resources of the current high‐performance cluster. Therefore, we propose communication‐efficient sparse multinomial logistic regression (CESMLR), which adopts the efficient network strategy of each node to solve the SMLR subproblem and achieve a large number of data partitions, taking full advantage of the computing resources of the cluster to achieve an efficient SMLR solution. The big data experimental results show that the performance of our algorithm exceeds those of state‐of‐the‐art algorithms. CESMLR is suitable for processing tasks with high‐dimensional features and consumes less running time while maintaining high classification accuracy.

EEG-based Classification of Lower Limb Motor Imagery with Brain Network Analysis

This study aims to investigate the difference in cortical signal characteristics between the left and right foot imaginary movements and to improve the classification accuracy of the experimental tasks. Raw signals were gathered from 64-channel scalp electroencephalograms of 11 healthy participants. Firstly, the cortical source model was defined with 62 regions of interest over the sensorimotor cortex (nine Brodmann areas). Secondly, functional connectivity was calculated by phase lock value for α and β rhythm networks. Thirdly, network-based statistics were applied to identify whether there existed stable and significant subnetworks that formed between the two types of motor imagery tasks. Meanwhile, ten graph theory indices were investigated for each network by t-test to determine statistical significance between tasks. Finally, sparse multinomial logistic regression (SMLR)-support vector machine (SVM), as a feature selection and classification model, was used to analyze the graph theory features. The specific time–frequency (α event-related desynchronization and β event-related synchronization) difference network between the two tasks was congregated at the midline and demonstrated significant connections in the premotor areas and primary somatosensory cortex. A few of statistically significant differences in the network properties were observed between tasks in the α and β rhythm. The SMLR-SVM classification model achieved fair discrimination accuracy between imaginary movements of the two feet (maximum 75% accuracy rate in single-trial analyses). This study reveals the network mechanism of the discrimination of the left and right foot motor imagery, which can provide a novel avenue for the BCI system by unilateral lower limb motor imagery.

A generic coordinate descent solver for non-smooth convex optimisation

ABSTRACT We present a generic coordinate descent solver for the minimisation of a non-smooth convex objective with structure. The method can deal in particular with problems with linear constraints. The implementation makes use of efficient residual updates and automatically determines which dual variables should be duplicated. A list of basic functional atoms is pre-compiled for efficiency and a modelling language in Python allows the user to combine them at run time. So, the algorithm can be used to solve a large variety of problems including Lasso, sparse multinomial logistic regression, linear and quadratic programmes.

Coupled Higher-Order Tensor Factorization for Hyperspectral and LiDAR Data Fusion and Classification

Hyperspectral and light detection and ranging (LiDAR) data fusion and classification has been an active research topic, and intensive studies have been made based on mathematical morphology. However, matrix-based concatenation of morphological features may not be so distinctive, compact, and optimal for classification. In this work, we propose a novel Coupled Higher-Order Tensor Factorization (CHOTF) model for hyperspectral and LiDAR data classification. The innovative contributions of our work are that we model different features as multiple third-order tensors, and we formulate a CHOTF model to jointly factorize those tensors. Firstly, third-order tensors are built based on spectral-spatial features extracted via attribute profiles (APs). Secondly, the CHOTF model is defined to jointly factorize the multiple higher-order tensors. Then, the latent features are generated by mode-n tensor-matrix product based on the shared and unshared factors. Lastly, classification is conducted by using sparse multinomial logistic regression (SMLR). Experimental results, conducted with two popular hyperspectral and LiDAR data sets collected over the University of Houston and the city of Trento, respectively, indicate that the proposed framework outperforms the other methods, i.e., different dimensionality-reduction-based methods, independent third-order tensor factorization based methods, and some recently proposed hyperspectral and LiDAR data fusion and classification methods.

Digital Mapping of Soil Classes Using Ensemble of Models in Isfahan Region, Iran

Digital soil maps can be used to depict the ability of soil to fulfill certain functions. Digital maps offer reliable information that can be used in spatial planning programs. Several broad types of data mining approaches through Digital Soil Mapping (DSM) have been tested. The usual approach is to select a model that produces the best validation statistics. However, instead of choosing the best model, it is possible to combine all models realizing their strengths and weaknesses. We applied seven different techniques for the prediction of soil classes based on 194 sites located in Isfahan region. The mapping exercise aims to produce a soil class map that can be used for better understanding and management of soil resources. The models used in this study include Multinomial Logistic Regression (MnLR), Artificial Neural Networks (ANN), Support Vector Machine (SVM), Decision Tree (DT), Random Forest (RF), Bayesian Networks (BN), and Sparse Multinomial Logistic Regression (SMnLR). Two ensemble models based on majority votes (Ensemble.1) and MnLR (Ensemble.2) were implemented for integrating the optimal aspects of the individual techniques. The overall accuracy (OA), Cohen's kappa coefficient index (κ) and the area under the curve (AUC) were calculated based on 10-fold-cross validation with 100 repeats at four soil taxonomic levels. The Ensemble.2 model was able to achieve larger OA, κ coefficient and AUC compared to the best performing individual model (i.e., RF). Results of the ensemble model showed a decreasing trend in OA from Order (0.90) to Subgroup (0.53). This was also the case for the κ statistic, which was the largest for the Order (0.66) and smallest for the Subgroup (0.43). Same decrease was observed for AUC from Order (0.81) to Subgroup (0.67). The improvement in κ was substantial (43 to 60%) at all soil taxonomic levels, except the Order level. We conclude that the application of the ensemble model using the MnLR was optimal, as it provided a highly accurate prediction for all soil taxonomic levels over and above the individual models. It also used information from all models, and thus this method can be recommended for improved soil class modelling. Soil maps created by this DSM approach showed soils that are prone to degradation and need to be carefully managed and conserved to avoid further land degradation.

Approximate Sparse Multinomial Logistic Regression for Classification.

We propose a new learning rule for sparse multinomial logistic regression (SMLR). The new rule is the generalization of the one proposed in the pioneering work by Krishnapuram et al. In our proposed method, the parameters of SMLR are iteratively estimated from log-posterior by using some approximations. The proposed update rule provides a faster convergence compared to the state-of the-art methods used for SMLR parameter estimation. The estimated parameters are tested on the pixel-based classification of hyperspectral images. The experimental results on real hyperspectral images show that the classification accuracy of proposed method is also better than those of the state-of-the-art methods.

Distributed Parallel Sparse Multinomial Logistic Regression

Sparse Multinomial Logistic Regression (SMLR) is widely used in the field of image classification, multi-class object recognition, and so on, because it has the function of embedding feature selection during classification. However, it cannot meet the time and memory requirements for processing large-scale data. We have reinvestigated the classification accuracy and running efficiency of the algorithm for solving SMLR problems using the Alternating Direction Method of Multipliers (ADMM), which is called fast SMLR (FSMLR) algorithm in this paper. By reformulating the optimization problem of FSMLR, we transform the serial convex optimization problem to the distributed convex optimization problem, i.e., global consensus problem and sharing problem. Based on the distributed optimization problem, we propose two distribute parallel SMLR algorithms, sample partitioning-based distributed SMLR (SP-SMLR), and feature partitioning-based distributed SMLR (FP-SMLR), for a large-scale sample and large-scale feature datasets in big data scenario, respectively. The experimental results show that the FSMLR algorithm has higher accuracy than the original SMLR algorithm. The big data experiments show that our distributed parallel SMLR algorithms can scale for massive samples and large-scale features, with high precision. In a word, our proposed serial and distribute SMLR algorithms outperform the state-of-the-art algorithms.

Shape-Adaptive Tensor Factorization Model for Dimensionality Reduction of Hyperspectral Images

Tensor-based dimensionality reduction (DR) of hyperspectral images is a promising research topic. However, patch-based tensorization usually adopts a squared neighborhood with fixed window size, which may be inaccurate in modeling the local spatial information in a hyperspectral image scene. In this work, we propose a novel shape-adaptive tensor factorization (SATF) model for dimensionality reduction and classification of hyperspectral images. Firstly, shape-adaptive patch features are extracted to build fourth-order tensors. Secondly, multilinear singular value decomposition (MLSVD) is adopted for tensor factorization and latent features are extracted via mode-i tensor-matrix product. Finally, classification is conducted by using a sparse multinomial logistic regression (SMLR) model. Experimental results, conducted with two popular hyperspectral data sets collected over the Indian Pines and the University of Pavia, respectively, indicate that the proposed method outperforms the other traditional and tensor-based DR methods.

2011. Identification of Streptococcus agalactiae on Human Fetal Membrane Tissues Using Raman Microspectroscopy

Background Streptococcus agalactiae, also known as Group B Streptococcus (GBS), colonizes 10–40% of women during late pregnancy and is an important cause of chorioamnionitis, or infection of the fetal membranes, and neonatal sepsis. The CDC recommends third trimester rectovaginal GBS screening, and intrapartum antibiotic prophylaxis for those testing positive. A rapid GBS diagnostic test could provide opportunities to identify GBS colonized women at the time of labor and focus the use of antibiotic therapy. Raman spectroscopy (RS) is an inelastic light scattering technique that provides biochemical spectra and has been used in vitro to characterize bacteria at the genus and species level. This study evaluated RS to identify and differentiate GBS, Escherichia coli, and Staphylococcus aureus ex vivo infection of human fetal membrane tissues.MethodsBacterial colonies of GBS, S. aureus, and E. coli were cultured on Mueller–Hinton agar. In addition, de-identified human fetal membrane tissues (VUMC IRB Approval #131607) were isolated and infected with 106 bacterial cells per 12 mm tissue punch for 48–72 hours. Samples from both were characterized using a Raman microscope. Hierarchical cluster analysis was implemented to evaluate principal component scores of Raman spectra from bacterial colonies. For tissue spectra, a machine learning algorithm, sparse multinomial logistic regression (SMLR), was used to determine the ability to discriminate across tissues types and identify biochemical features important for classification. Following RS analysis, scanning electron microscopy was performed to verify the presence of bacterial cells at the site of Raman measurements.ResultsUnique spectral features were identified from colonies grown on agar and infected fetal membrane tissues. Analysis using SMLR accurately identified GBS-infected tissues with 92.2% sensitivity and specificity. Scanning electron microscopy evaluation confirmed the presence of bacterial cells that were structured in biofilms at the site of Raman measurements.ConclusionTogether, these findings support further investigation into the use of RS as an emerging microbiologic diagnostic tool and intrapartum screening test for GBS carriage. Disclosures All authors: No reported disclosures.

F-score feature selection based Bayesian reconstruction of visual image from human brain activity

Abstract Decoding perceptual experience from human brain activity is a big challenge in neuroscience. Recent advances in human neuroimaging have shown that it is possible to reconstruct a person’s visual experience based on the retinotopy in the early visual cortex and the multivariate pattern analysis (MVPA) method using functional magnetic resonance imaging (fMRI). Previous researches reconstructed binary contrast-defined images using combination of multi-scale local image decoders in V1, V2 and V3, where contrast for local image bases was predicted from fMRI activity by sparse multinomial logistic regression (SMLR) and other models. However, the precision and efficiency of the visual image reconstruction remain insufficient. Proper feature selection is widely known to be as critical for prediction and reconstruction. Aiming at the shortcomings of existing reconstruction models, we proposed a new model of Bayesian reconstruction based on F-score feature selection (Bayes+F). The results indicate that the proposed Bayes+F model has better reconstruction accuracy and higher efficiency than the SMLR and other models, showing better robustness and noise resistant ability. It can improve the spatial correlation coefficient (Mean ± variance: 0.7078 ± 0.2104) and decrease the standard error (Mean ± variance: 0.2693 ± 0.0871) between the stimulus and the reconstructed image. Furthermore, the proposed model can reconstruct the images extremely rapid, 100 times faster than SMLR does.

Discrimination of malignant and normal kidney tissue with short wave infrared dispersive Raman spectroscopy.

Renal mass biopsy is still controversial due to imperfect accuracy. Raman spectroscopy (RS) demonstrated promise as an in vivo real-time, nondestructive diagnostic tool in many malignancies. Short wave infrared (SWIR) RS has the potential to improve on previous RS systems for renal mass diagnosis. The aim of this study is to evaluate a SWIR RS system in differentiating normal and malignant renal samples. Measurements were acquired using a benchtop RS system with excitation wavelength at 1064 nm and an InGaAs array detector. Processed spectra were classified with a Bayesian machine learning algorithm, sparse multinomial logistic regression. Sensitivity and receiver operating characteristic curve analyses evaluated the classifier accuracy. Accuracy of the classifier was 92.5% with sensitivity and specificity of 95.8% and 88.8%, respectively. For posterior probability of malignant class assignment, the area under the ROC curve is 0.94 (95% confidence interval: 0.89-0.99, P < .001). SWIR RS accurately differentiated normal and malignant kidney tumors. RS has the potential to be used as a diagnostic tool in kidney cancer.

GPU Parallel Implementation of Spatially Adaptive Hyperspectral Image Classification

Image classification is a very important tool for remotely sensed hyperspectral image processing. Techniques able to exploit the rich spectral information contained in the data, as well as its spatial-contextual information, have shown success in recent years. Due to the high dimensionality of hyperspectral data, spectral-spatial classification techniques are quite demanding from a computational viewpoint. In this paper, we present a computationally efficient parallel implementation for a spectral-spatial classification method based on spatially adaptive Markov random fields (MRFs). The method learns the spectral information from a sparse multinomial logistic regression classifier, and the spatial information is characterized by modeling the potential function associated with a weighted MRF as a spatially adaptive vector total variation function. The parallel implementation has been carried out using commodity graphics processing units (GPUs) and the NVIDIA's Compute Unified Device Architecture. It optimizes the work allocation and input/output transfers between the central processing unit and the GPU, taking full advantages of the computational power of GPUs as well as the high bandwidth and low latency of shared memory. As a result, the algorithm exploits the massively parallel nature of GPUs to achieve significant acceleration factors (higher than 70x) with regards to the serial and multicore versions of the same classifier on an NVIDIA Tesla K20C platform.

A novel SMLR-PSO model to estimate the chlorophyll content in the crops using hyperspectral satellite images

The estimating the chlorophyll contents in the crops helps to identify the condition of crops and different classification of crops with soil characteristics in order to assist the farmer or others with agriculture growth. In this paper, a hybrid approach is introduced to estimate the Chlorophyll contents in the crops using hyperspectral image segmentation with active learning, which consists of two main steps. First, we use a sparse multinomial logistic regression (SMLR) model to learn the class posterior probability distributions with Quadratic Programming or joint probability distribution. Second, we use the information acquired in the previous step to segment the hyper spectral image using a Markov Random field segments to estimate the dependencies using spatial information and edge Information by minimum spanning forest rooted on markers. In order to reduce the cost of acquiring large training sets, PSO optimization is performed based on the SMLR posterior probabilities on the Normalized difference vegetation index (NDVI). The state-of-the-art performance of the proposed approach is illustrated using real hyper spectral data sets collected from the North Karnataka in a number of experimental comparisons with recently developed or statistical hyperspectral image analysis methods in terms of precision, recall and f—measure.