Variational Bayesian Matrix Factorization Research Articles

Deep Convolutional Neural Network Compression Method: Tensor Ring Decomposition with Variational Bayesian Approach

Due to deep neural networks (DNNs) a large number of parameters, DNNs increase the demand for computing and storage during training, reasoning and deployment, especially when DNNs stack deeper and wider. Tensor decomposition can not only compress DNN models but also reduce parameters and storage requirements while maintaining high accuracy and performance. About tensor ring (TR) decomposition of tensor decomposition, there are two problems: (1) The practice of setting the TR rank to be equal in TR decomposition results in an unreasonable rank configuration. (2) The training time of selecting rank through iterative processes is time-consuming. To address the two problems, a TR network compression method by Variational Bayesian (TR-VB) is proposed based on the Global Analytic Solution of Empirical Variational Bayesian Matrix Factorization (GAS of EVBMF). The method consists of three steps: (1) rank selection, (2) TR decomposition, and (3) fine-tuning to recover accumulated loss of accuracy. Experimental results show that, for a given network, TR-VB gives the best results in terms of Top-1 accuracy, parameters, and training time under different compression levels. Furthermore, TR-VB validated on CIFAR-10/100 public benchmarks achieves state-of-the-art performance.

A Holistic Strategy for Classification of Sleep Stages with EEG.

Manual sleep stage scoring is usually implemented with the help of sleep specialists by means of visual inspection of the neurophysiological signals of the patient. As it is a very hectic task to perform, automated sleep stage classification systems were developed in the past, and advancements are being made consistently by researchers. The various stages of sleep are identified by these automated sleep stage classification systems, and it is quite an important step to assist doctors for the diagnosis of sleep-related disorders. In this work, a holistic strategy named as clustering and dimensionality reduction with feature extraction cum selection for classification along with deep learning (CDFCD) is proposed for the classification of sleep stages with EEG signals. Though the methodology follows a similar structural flow as proposed in the past works, many advanced and novel techniques are proposed under each category in this work flow. Initially, clustering is applied with the help of hierarchical clustering, spectral clustering, and the proposed principal component analysis (PCA)-based subspace clustering. Then the dimensionality of it is reduced with the help of the proposed singular value decomposition (SVD)-based spectral algorithm and the standard variational Bayesian matrix factorization (VBMF) technique. Then the features are extracted and selected with the two novel proposed techniques, such as the sparse group lasso technique with dual-level implementation (SGL-DLI) and the ridge regression technique with limiting weight scheme (RR-LWS). Finally, the classification happens with the less explored multiclass Gaussian process classification (MGC), the proposed random arbitrary collective classification (RACC), and the deep learning technique using long short-term memory (LSTM) along with other conventional machine learning techniques. This methodology is validated on the sleep EDF database, and the results obtained with this methodology have surpassed the results of the previous studies in terms of the obtained classification accuracy reporting a high accuracy of 93.51% even for the six-classes classification problem.

Classification of cardiac diseases from ECG signals through bio inspired classifiers with Adam and R-Adam approaches for hyperparameters updation

This study emphasizes the classification of different cardiac diseases through bio-inspired classifiers with and without hyperparameters selection. Principal Component Analysis (PCA), Linearity Preserving Projection (LPP), Kernel-Linear Discriminant Analysis (K-LDA), and Variational Bayesian Matrix Factorization (VBMF) are used for the dimensionality reduction of ECG Signals. After the dimensionality reduction, the ECG features were classified using several classifiers such as Particle Swarm Optimization (PSO), Fish Swarm Optimization (FSO), Whale Optimization Algorithm (WOA), Grey Wolf Optimization (GWO), Support Vector Machine with (RBF (kernel)), K Nearest Neighbor (KNN) and Naïve Bayesian Classifier (NBC). The average accuracy for the classifiers without hyperparameters selection is 64.99%. Further, the classifiers (PSO, FSO, WOA, and GWO) performance is enhanced through the selection of hyperparameters by the Adam and Randomized Adam (R-Adam) approaches. The average accuracy for the classifiers with Adam and R-Adam is at 72.32% and 85.63%. In this research, the MIT-BIH datasets of Atrial Fibrillation (AF), Premature Ventricular Contraction (PVC), ST Change (ST), Supra Ventricular Tachycardia (SVT), Ventricular Tachycardia (VT) and Normal Sinus Rhythm (NSR) are utilized from the physionet archived database. The Overall Accuracy (OA), Mean Square Error (MSE), F1-Score (F1), Fowlkes Mallows Index (FM), Mathew Correlation Coefficient (MCC) and Error Rate (ER) of the different classifiers with and without Adam and R-Adam methods are evaluated. The result demonstrates that the GWO-R-Adam classifier with the LPP dimensionality reduction technique with 98.90% overall accuracy in classifying {ST}-{NSR} classes outperforms all other classifiers in terms of classification performance measures.

Improved Sparse Representation based Robust Hybrid Feature Extraction Models with Transfer and Deep Learning for EEG Classification

Numerous studies in the field of cognitive research is dependent on Electroencephalography (EEG) as it apprehends the neural correspondences of various mental activities such as memory, speech, or attention related activities with millisecond precision. In this work, initially sparse rendition is imposed on EEG signals and then sparse optimization is performed to it. For sparse representation, initially the work utilized both K-Singular Value Decomposition (K-SVD) for dictionary learning along with Orthogonal Matching Pursuit (OMP) for sparse coding. Later an advanced OMP technique was developed which made a significant improvement, and then instead of K-SVD, K-means and Method of Optimal direction (MOD) techniques were utilized with it and as a result, totally six different combinations in sparse representation optimization were developed. Then, it is modelled into clusters using seven different hybrid models developed here for the purpose of feature extraction and selection. Out of the seven models, two models are already existing such as Finite Mixture Logistic Regression Model (FMLR) and Expectation-Maximization based Gaussian Mixture Model (EM-GMM) which are widely used. The remaining five hybrid models are proposed as variations of GMM such as Variational Bayesian Matrix Factorization (VBMF) based GMM (VBMF-GMM), VBMF and Probabilistic Principal Component Analysis (PCA) with GMM (VBMF-PCA-GMM), VBMF and Partial Least Squares (PLS) with GMM (VBMF-PLS-GMM), VBMF and Canonical Correlation Analysis (CCA) with GMM (VBMF-CCA-GMM) and Empirical VBMF with GMM (EVBMF-GMM). This model is validated on two unique EEG datasets, such as epilepsy dataset and schizophrenia dataset, and the results have been classified with the standard machine learning techniques, transfer learning techniques and the proposed deep learning techniques leading to a total of twelve different classifiers and a comprehensive analysis is made in this work with very promising results in terms of classification accuracy reporting more than 95% for most of the proposed cases.

Variational autoencoder Bayesian matrix factorization (VABMF) for collaborative filtering

Probabilistic matrix factorization (PMF) is the most popular method among low-rank matrix approximation approaches that address the sparsity problem in collaborative filtering for recommender systems. PMF depends on the classical maximum a posteriori estimator for estimating model parameters; however, these approaches are vulnerable to overfitting because of the nature of a single point estimation that is pursued by these models. An alternative approach to PMF is a Bayesian PMF model that suggests the Markov chain Monte Carlo algorithm as a full estimation for approximate intractable posterior over model parameters. However, despite its success in increasing prediction, it has a high computational cost. To this end, we proposed a novel Bayesian deep learning-based model treatment, namely, variational autoencoder Bayesian matrix factorization (VABMF). The proposed model uses stochastic gradient variational Bayes to estimate intractable posteriors and expectation–maximization-style estimators to learn model parameters. The model was evaluated on the basis of three MovieLens datasets, namely, Ml-100k, Ml-1M, and Ml-10M. Experimental results showed that our proposed VABMF model significantly outperforms state-of-the-art RS.

Bayesian Optimization-Based Global Optimal Rank Selection for Compression of Convolutional Neural Networks

Recently, convolutional neural network (CNN) compression via low-rank decomposition has achieved remarkable performance. Finding the optimal rank is a crucial problem because rank is the only hyperparameter for controlling computational complexity and accuracy in compressed CNNs. In this paper, we propose a global optimal rank selection method based on Bayesian optimization (BayesOpt), which is a machine learning based global optimization technique. By utilizing both a simple objective function and a proper optimization scheme, the proposed method produces a global optimal rank that provides a good trade-off between computational complexity and accuracy degradation. In addition, our method also reflects the correlation of each rank in multi-rank selection, and is able to flexibly yield an optimal rank with a given fixed compression ratio. Experimental results indicate that the proposed algorithm can identify the global optimal rank regardless of the huge size of dataset or the various structural features of CNNs. In all experiments on multi-rank selection, the proposed method produces the rank with higher accuracy and lower computational complexity than the state-of-the-art rank selection method, variational Bayesian matrix factorization (VBMF).

VB-MK-LMF: fusion of drugs, targets and interactions using variational Bayesian multiple kernel logistic matrix factorization

BackgroundComputational fusion approaches to drug-target interaction (DTI) prediction, capable of utilizing multiple sources of background knowledge, were reported to achieve superior predictive performance in multiple studies. Other studies showed that specificities of the DTI task, such as weighting the observations and focusing the side information are also vital for reaching top performance.MethodWe present Variational Bayesian Multiple Kernel Logistic Matrix Factorization (VB-MK-LMF), which unifies the advantages of (1) multiple kernel learning, (2) weighted observations, (3) graph Laplacian regularization, and (4) explicit modeling of probabilities of binary drug-target interactions.ResultsVB-MK-LMF achieves significantly better predictive performance in standard benchmarks compared to state-of-the-art methods, which can be traced back to multiple factors. The systematic evaluation of the effect of multiple kernels confirm their benefits, but also highlights the limitations of linear kernel combinations, already recognized in other fields. The analysis of the effect of prior kernels using varying sample sizes sheds light on the balance of data and knowledge in DTI tasks and on the rate at which the effect of priors vanishes. This also shows the existence of “small sample size” regions where using side information offers significant gains. Alongside favorable predictive performance, a notable property of MF methods is that they provide a unified space for drugs and targets using latent representations. Compared to earlier studies, the dimensionality of this space proved to be surprisingly low, which makes the latent representations constructed by VB-ML-LMF especially well-suited for visual analytics. The probabilistic nature of the predictions allows the calculation of the expected values of hits in functionally relevant sets, which we demonstrate by predicting drug promiscuity. The variational Bayesian approximation is also implemented for general purpose graphics processing units yielding significantly improved computational time.ConclusionIn standard benchmarks, VB-MK-LMF shows significantly improved predictive performance in a wide range of settings. Beyond these benchmarks, another contribution of our work is highlighting and providing estimates for further pharmaceutically relevant quantities, such as promiscuity, druggability and total number of interactions.

ASSESSMENT OF EPILEPSY CLASSIFICATION USING TECHNIQUES SUCH AS SINGULAR VALUE DECOMPOSITION, APPROXIMATE ENTROPY, AND WEIGHTED K-NEAREST NEIGHBORS MEASURES

ABSTRACTObjective: The main aim of this research is to reduce the dimension of the epileptic Electroencephalography (EEG) signals and then classify it usingvarious post classifiers. For the evaluation and easy treatment of neurological diseases, EEG signals are used. The reflection of the electrical activitiesof the human brain is obtained by the measurement of potentials in EEG. To study and explore the brain functions in an exhaustive manner, EEG is usedby both physicians and scientists. The study of the electrical activity of the brain which is done through EEG recording is a vital tool for the diagnosis ofmany neurological diseases which include epilepsy, sleep disorders, injuries in head, dementia etc. One of the most commonly occurring and prevalentneurological disorders is epilepsy and it is easily characterized by recurrent seizures.Methods: This paper employs the concept of dimensionality reduction concepts like Fuzzy Mutual Information (FMI), Independent ComponentAnalysis (ICA), Linear Graph Embedding (LGE), Linear Discriminant Analysis (LDA) and finally Variational Bayesian Matrix Factorization (VBMF).The epilepsy risk levels are also classified using post classifiers like Singular Value Decomposition (SVD), Approximate Entropy (ApEn) and WeightedKNN (W-KNN) classifiers.Results: The highest accuracy is obtained when LDA is combined with Weighted KNN (W-KNN) Classifiers and it is of 97.18%. Conclusion: Thus the EEG signals not only represent the brain function but also the status of the whole body. The best result obtained was whenLDA is engaged as a dimensionality reduction technique followed by the usage of the W-KNN as post classifier for the classification of epilepsy risklevels from EEG signals. Future work may incorporate the possible usage of different dimensionality reduction techniques with various other types ofclassifiers for the perfect classification of epilepsy risk levels from EEG signals.Keywords: FMI, ICA, LGE, LDA, W-KNN, EEG

A Unique Approach to Epilepsy Classification from EEG Signals Using Dimensionality Reduction and Neural Networks

Characterized by recurrent and rapid seizures, epilepsy is a great threat to the livelihood of the human beings. Abnormal transient behaviour of neurons in the cortical regions of the brain leads to a seizure which characterizes epilepsy. The physical and mental activities of the patient are totally dampened with this epileptic seizure. A significant clinical tool for the study, analysis and diagnosis of the epilepsy is electroencephalogram (EEG). To detect such seizures, EEG signals aids greatly to the clinical experts and it is used as an important tool for the analysis of brain disorders, especially epilepsy. In this paper, the high dimensional EEG data are reduced to a low dimension by incorporating techniques such as Fuzzy Mutual Information (FMI), Independent Component Analysis (ICA), Linear Graph Embedding (LGE), Linear Discriminant Analysis (LDA) and Variational Bayesian Matrix Factorization (VBMF). After employing them as dimensionality reduction techniques, the Neural Networks (NN) such as Cascaded Feed Forward Neural Network (CFFNN), Time Delay Neural Network (TDNN) and Generalized Regression Neural Network (GRNN) are used as Post Classifiers for the Classification of Epilepsy Risk Levels from EEG signals. The bench mark parameters used here are Performance Index (PI), Quality Values (QV), Time Delay, Accuracy, Specificity and Sensitivity.

Variational Bayesian Matrix Factorization and Certain Post Classifiers for Classification of Epilepsy from EEG Signals

The main aim of this paper is to employ Variational Bayesian Matrix Factorization (VBMF) as a dimensionality reduction technique followed by the Gaussian Mixture Model (GMM), Genetic Algorithm (GA) and Naive Bayes Classifier (NBC) as post classifiers for the classification of epilepsy risk levels from Electroencephalography (EEG) Signals. Since epilepsy is one of the serious disorders of the brain which is characterized by frequent and recurrent seizures, the detection and classification of it seems to be very important. Using the EEG signals, the epileptic seizures can be analyzed because it aids in the recording, diagnosing and for treating other neurological disorders. In this paper, the results are analyzed and compared in terms of sensitivity, specificity, time delay, quality values, performance index and accuracy.

Variational Bayesian Matrix Factorization for Bounded Support Data.

A novel Bayesian matrix factorization method for bounded support data is presented. Each entry in the observation matrix is assumed to be beta distributed. As the beta distribution has two parameters, two parameter matrices can be obtained, which matrices contain only nonnegative values. In order to provide low-rank matrix factorization, the nonnegative matrix factorization (NMF) technique is applied. Furthermore, each entry in the factorized matrices, i.e., the basis and excitation matrices, is assigned with gamma prior. Therefore, we name this method as beta-gamma NMF (BG-NMF). Due to the integral expression of the gamma function, estimation of the posterior distribution in the BG-NMF model can not be presented by an analytically tractable solution. With the variational inference framework and the relative convexity property of the log-inverse-beta function, we propose a new lower-bound to approximate the objective function. With this new lower-bound, we derive an analytically tractable solution to approximately calculate the posterior distributions. Each of the approximated posterior distributions is also gamma distributed, which retains the conjugacy of the Bayesian estimation. In addition, a sparse BG-NMF can be obtained by including a sparseness constraint to the gamma prior. Evaluations with synthetic data and real life data demonstrate the good performance of the proposed method.

Variational Bayesian View of Weighted Trace Norm Regularization for Matrix Factorization

Matrix factorization with trace norm regularization is a popular approach to matrix completion and collaborative filtering. When entries of the matrix are sampled non-uniformly (which is the case for collaborative prediction), a properly weighted correction to the trace norm regularization is known to improve the performance dramatically. While the weighted trace norm regularization has been rigorously studied, its generative counterpart is not known yet. In this paper we show that the weighted trace norm regularization emerges from variational Bayesian matrix factorization where variational distributions over factor matrices are restricted to be isotropic Gaussians with the common variance. We show that variational variance corresponds to the regularization parameter. Thus, the regularization parameter can be automatically learned by variational inference rather than cross-validation. Experiments on MovieLens and Netflix datasets confirm the variational Bayesian perspective of the weighted trace norm regularization, demonstrating that variational parameter learned by variational inference agrees with the value of the regularization parameter found by cross-validation.

Global analytic solution of fully-observed variational Bayesian matrix factorization

The variational Bayesian (VB) approximation is known to be a promising approach to Bayesian estimation, when the rigorous calculation of the Bayes posterior is intractable. The VB approximation has been successfully applied to matrix factorization (MF), offering automatic dimensionality selection for principal component analysis. Generally, finding the VB solution is a nonconvex problem, and most methods rely on a local search algorithm derived through a standard procedure for the VB approximation. In this paper, we show that a better option is available for fully-observed VBMF--the global solution can be analytically computed. More specifically, the global solution is a reweighted SVD of the observed matrix, and each weight can be obtained by solving a quartic equation with its coefficients being functions of the observed singular value. We further show that the global optimal solution of empirical VBMF (where hyperparameters are also learned from data) can also be analytically computed. We illustrate the usefulness of our results through experiments in multi-variate analysis.