Sparse Learning Framework Research Articles

Deep discriminative sparse representation learning for machinery fault diagnosis

The high complexity of actual machinery vibration environments introduces various interferences into vibration signals, making it challenging to eliminate redundant information and extract discriminative features of mechanical faults. Shallow-structured sparse classification models enable intelligent fault diagnosis but struggle with deeper feature extraction, especially in small-sample cases. In this paper, we introduce a deep discriminative sparse representation learning (DDSRL) framework with a deep architecture for this issue. In DDSRL, the first-layer dictionary atoms are automatically learned from raw signals and borrowed for sparse coding to enhance sparsity and interclass discriminability. Then, the weight matrix is defined to screen high-energy atoms from the upper dictionary as new training samples for the next dictionary learning and sparse coding layers, thereby eliminating interclass redundant atoms and mining deeper discriminative representations of cyclostationary impulses or meshing harmonics from the shallow dictionary. Finally, DDSRL extracts feature sequences sensitive to fault-induced waveforms via the learned dictionaries and coding coefficients for intelligent fault identification. Experimental verifications on two datasets with high-speed bearing and compound gear-bearing faults demonstrate that DDSRL outperforms six popular sparse classification models, sparse autoencoder, convolutional neural network, and principal component analysis network. Furthermore, the layer-by-layer feature extraction process of DDSRL and feature visualization analysis provide more credible fault diagnosis results.

Sparsify dynamically expandable network via variational dropout

This paper develops a new method for lifelong learning referred to as Sparsify Dynamically Expandable Network (SDEN) via Variational Dropout, which explores a sparse model while preserving the performance. Dynamically Expandable Network (DEN) can learn a sequence of tasks via performing network retraining, network expansion by adding only the necessary neurons, and network split to effectively prevent semantic drift in an online manner. To overcome point estimation of parameters in DEN, Bayesian Compression for DEN is developed under the Bayesian framework. However, this method demands more time for model training and testing. To improve the model efficiency, we propose SDEN under the efficient sparse learning framework. We validate our SDEN in the lifelong learning scenarios with multiple frequently used benchmarks, on which it can obtain comparable classification accuracy, and less training and testing time compared with the comparison methods. Furthermore, our method can also learn a more sparse network structure which means fewer network parameters.

Towards Effective and Efficient Sparse Neural Information Retrieval

Sparse representation learning based on Pre-trained Language Models has seen a growing interest in Information Retrieval. Such approaches can take advantage of the proven efficiency of inverted indexes and inherit desirable IR priors such as explicit lexical matching or some degree of interpretability. In this work, we thoroughly develop the framework of sparse representation learning in IR, which unifies term weighting and expansion in a supervised setting. We then build on SPLADE—a sparse expansion-based retriever—and show to which extent it is able to benefit from the same training improvements as dense bi-encoders by studying the effect of distillation, hard negative mining, as well as the Pre-trained Language Model’s initialization on its effectiveness , leading to state-of-the-art results in both in- and out-of-domain evaluation settings (SPLADE++). We furthermore propose efficiency improvements, allowing us to reach latency requirements on par with traditional keyword-based approaches (Efficient-SPLADE).

Fast sparse twin learning framework for large-scale pattern classification

Lately, twin support vector machine and its variants have received extensive attention and in-depth research in the field of large-scale pattern classification. However, they may lead to high computational cost, which greatly affects their development and research. Aiming to solve this problem, this paper a novel fast sparse twin learning framework for large-scale data classification is proposed. In this learning framework, by introducing sparse constraints into the dual problem, the number of support vectors can be effectively reduced in dual space, so as to improve the computational speed of the model. Importantly, the learning framework is not only sparsity for large-scale samples classification, but also insensitive to sample noise, thus it is stable for resampling. In addition, we use the modified Newton’s method to deal with the optimization problem with sparse constraints. Numerical experiments are carried out on ten large-scale datasets. The results show that the proposed learning framework for large-scale data classification problems has significant advantages and comparability with other learning methods in terms of computational speed and classification accuracy.

PointSGRADE: Sparse learning with graph representation for anomaly detection by using unstructured 3D point cloud data

Surface anomaly detection by using 3D point cloud data has recently received significant attention. To completely measure the common free-form surfaces without loss of details, advanced 3D scanning technologies, such as 3D laser scanners, can be applied and will produce an unstructured point cloud. However, this irregular data structure poses challenges to anomaly detection, in that the existing methods based on regular data, e.g., 2D image, cannot be directly applied. This article proposes a sparse learning framework with a graph representation of the unstructured point cloud for anomaly detection (PointSGRADE). Specifically, the free-form surface is assumed to be smooth. Then, the associated point cloud can be represented as a graph. Subsequently, considering the sparse anomalies, we propose a sparse learning framework and formulate the anomaly detection problem as a penalized optimization problem, which is further solved by a computationally efficient majorization-minimization framework. Case studies demonstrate the accuracy and robustness of the proposed method. This article proposes a novel methodology for sparse anomaly detection on smooth free-form surfaces represented by unstructured point cloud, which is critical for quality inspection in manufacturing and other application areas.

Optimal selection of dictionary atoms for sparse dictionary learning of time-varying monitoring data in two-dimensional geotechnical problems

Geotechnical monitoring data (e.g., settlement monitoring data) are often sparsely measured at limited locations, but the corresponding geotechnical quantities of interest (e.g., ground settlement) usually vary in both temporal and spatial dimensions. The sparsely measured monitoring data can be efficiently leveraged under a sparse dictionary learning (SDL) framework, which represents a spatio-temporally varying quantity (e.g., settlement) by a weighted summation of dictionary atoms. However, selection of dictionary atoms in SDL greatly affects the SDL performance, and under-fitting or over-fitting problems are frequently encountered, particularly when dealing with limited monitoring data. To tackle these problems, a knee point-based method is proposed in this study to properly select dictionary atoms for optimizing the SDL performance. The proposed method not only automatically determines dictionary atom IDs and weights, but also strikes a balance between the under-fitting and over-fitting. The proposed approach is illustrated using a real case history of embankments constructed on soft clay in Australia. The proposed approach significantly improves the prediction of time-varying settlements that also spatially vary over a two-dimensional (2D) space (e.g., a vertical cross-section), particularly at subsequent time steps and spatial locations without monitoring data. Effects of the dictionary atom quantity on settlement predictions are also investigated.

An effective cost-sensitive sparse online learning framework for imbalanced streaming data classification and its application to online anomaly detection

An effective cost-sensitive sparse online learning framework for imbalanced streaming data classification and its application to online anomaly detection

Robust dual-graph regularized and minimum redundancy based on self-representation for semi-supervised feature selection

Partial labeled data is ubiquitous in the big data era. Selecting informative features, and avoiding redundant and noise features is an important task for constructing robust learning models. The inherent characteristics of samples and features need to be surveyed simultaneously. In this study, a feature selection method for partial labeled data based on dual-graph regularized is proposed. Self-representation can well mine the relation between features. Then, the self-representation is combined with the framework of sparse learning to well embody the character of data and an adaptive redundancy regularization term based on self-representation is designed to minimize the redundancy between features. In the self-representation based regularization term, the relation between features is updated dynamically during the iteration so as to capture the inherent relation between features more accurately. The manifold structure in both feature space and data space are considered jointly. Moreover, the L2,p-norm is imposed on the feature self-representation regularization term, self-representation coefficient matrix, and feature selection matrix, respectively. These constrains aim to enhance its robustness to outliers, and select the representative features with discriminative and low redundancy. The convex (p=1) and non-convex (0<p<1) are involved in the proposed method. Then, a unified solution for the proposed method is investigated and its convergence is proved. Experimental results on public data sets show that the proposed method is effective for classification tasks when compared with some feature selection methods.

Discriminating mild traumatic brain injury using sparse dictionary learning of functional network dynamics.

Mild traumatic brain injury (mTBI) is usually caused by a bump, blow, or jolt to the head or penetrating head injury, and carries the risk of inducing cognitive disorders. However, identifying the biomarkers for the diagnosis of mTBI is challenging as evident abnormalities in brain anatomy are rarely found in patients with mTBI. In this study, we tested whether the alteration of functional network dynamics could be used as potential biomarkers to better diagnose mTBI. We propose a sparse dictionary learning framework to delineate spontaneous fluctuation of functional connectivity into the subject‐specific time‐varying evolution of a set of overlapping group‐level sparse connectivity components (SCCs) based on the resting‐state functional magnetic resonance imaging (fMRI) data from 31 mTBI patients in the early acute phase (<3 days postinjury) and 31 healthy controls (HCs). The identified SCCs were consistently distributed in the cohort of subjects without significant inter‐group differences in connectivity patterns. Nevertheless, subject‐specific temporal expression of these SCCs could be used to discriminate patients with mTBI from HCs with a classification accuracy of 74.2% (specificity 64.5% and sensitivity 83.9%) using leave‐one‐out cross‐validation. Taken together, our findings indicate neuroimaging biomarkers for mTBI individual diagnosis based on the temporal expression of SCCs underlying time‐resolved functional connectivity.

ParsVNN: parsimony visible neural networks for uncovering cancer-specific and drug-sensitive genes and pathways.

Prediction of cancer-specific drug responses as well as identification of the corresponding drug-sensitive genes and pathways remains a major biological and clinical challenge. Deep learning models hold immense promise for better drug response predictions, but most of them cannot provide biological and clinical interpretability. Visible neural network (VNN) models have emerged to solve the problem by giving neurons biological meanings and directly casting biological networks into the models. However, the biological networks used in VNNs are often redundant and contain components that are irrelevant to the downstream predictions. Therefore, the VNNs using these redundant biological networks are overparameterized, which significantly limits VNNs’ predictive and explanatory power. To overcome the problem, we treat the edges and nodes in biological networks used in VNNs as features and develop a sparse learning framework ParsVNN to learn parsimony VNNs with only edges and nodes that contribute the most to the prediction task. We applied ParsVNN to build cancer-specific VNN models to predict drug response for five different cancer types. We demonstrated that the parsimony VNNs built by ParsVNN are superior to other state-of-the-art methods in terms of prediction performance and identification of cancer driver genes. Furthermore, we found that the pathways selected by ParsVNN have great potential to predict clinical outcomes as well as recommend synergistic drug combinations.

Deep multiple-instance learning for abnormal cell detection in cervical histopathology images

Cervical cancer is a disease of significant concern affecting women's health worldwide. Early detection of and treatment at the precancerous stage can help reduce mortality. High-grade cervical abnormalities and precancer are confirmed using microscopic analysis of cervical histopathology. However, manual analysis of cervical biopsy slides is time-consuming, needs expert pathologists, and suffers from reader variability errors. Prior work in the literature has suggested using automated image analysis algorithms for analyzing cervical histopathology images captured with the whole slide digital scanners (e.g., Aperio, Hamamatsu, etc.). However, whole-slide digital tissue scanners with good optical magnification and acceptable imaging quality are cost-prohibitive and difficult to acquire in low and middle-resource regions. Hence, the development of low-cost imaging systems and automated image analysis algorithms are of critical importance. Motivated by this, we conduct an experimental study to assess the feasibility of developing a low-cost diagnostic system with the H&E stained cervical tissue image analysis algorithm. In our imaging system, the image acquisition is performed by a smartphone affixing it on the top of a commonly available light microscope which magnifies the cervical tissues. The images are not captured in a constant optical magnification, and, unlike whole-slide scanners, our imaging system is unable to record the magnification. The images are mega-pixel images and are labeled based on the presence of abnormal cells. In our dataset, there are total 1331 (train: 846, validation: 116 test: 369) images. We formulate the classification task as a deep multiple instance learning problem and quantitatively evaluate the classification performance of four different types of multiple instance learning algorithms trained with five different architectures designed with varying instance sizes. Finally, we designed a sparse attention-based multiple instance learning framework that can produce a maximum of 84.55% classification accuracy on the test set.

Statistical n-Best AFD-Based Sparse Representation for ECG Biometric Identification

Electrocardiogram (ECG) biometric recognition as a personal identification method is receiving more and more attention because it can support live verification results. Compared with other biometric-based methods, it can provide higher security performance. The difficulty of the problem lies in how to stably extract ECG signal features and achieve real-time verification. In this study, a new type of sparse representation learning framework called statistical $n$ -best adaptive Fourier decomposition (SAFD) originated by Qian is adopted in ECG biometric identification. Adaptive Fourier decomposition (AFD) is a recently developed combination of transform-based signal decomposition and sparse representation method, which can adaptively select the atoms from a redundant dictionary through orthogonal processing. The advantage of the AFD-type methods is that each atom in the dictionary has a precise mathematical formula with good analytic properties. This characteristic is significantly distinguished it from other existing sparse representations, where the atoms learned are usually matrix data and cannot be described mathematically. The proposed SAFD extends the existing $n$ -best AFD from processing single signal to multi-signals and implements the $n$ -best AFD in the stochastic Hardy space. Therefore, the small number of learned atoms by SAFD is sufficient to capture internal structure and robustness of the signal and generate a discriminative representation that reflects the time–frequency characteristics of signals. It is very suitable for non-stationary signals like ECG. The proof of convergence of the algorithm is presented. Extensive experiments are conducted on five public databases collected in different realistic conditions, and an average identification accuracy of 98.0% is achieved. In addition, less than 1 ms for one matching process makes it possible to be implemented in real time. Experimental results demonstrate that the proposed method can achieve superior performance compared to other state-of-the-art ECG biometric identification methods.

Nonnegative Bounded Convolutional Sparse Learning Method for Envelope Feature Deconvolution

This article considers the problem of extracting periodic envelope features from observation signals modulated by subcritically damped mechanical systems. Due to insufficient envelope structure descriptions, popular two-stage blind deconvolution techniques incur a significant performance degradation in envelope feature deconvolution tasks. Leveraging sparse learning framework, a nonnegative bounded convolutional sparse learning model (NBconvSLM), is then proposed to address it in this article, and meanwhile, a nonconvex multiblock alternating direction method of multiplier (ADMM) solver is developed to rapidly achieve satisfying deconvolutional solutions. The main highlight of NBconvSLM is that nonnegative sparse regularizer is exploited to effectively describe envelope feature structure, and bounded regularizer is further introduced to sufficiently mitigate deconvolutional envelope amplitude enlargement problem. Regularizer’ roles are verified through a set of numerical experiments. Meanwhile, algorithmic performance and superiority are profoundly evaluated with respect to state-of-the-art deconvolutional techniques. Engineering application to generator bearing fault detection further corroborates algorithmic effectiveness in latent periodic feature recognition.

Self-training and learning the waveform features of microseismic data using an adaptive dictionary

Microseismic monitoring is an indispensable technique in characterizing the physical processes that are caused by extraction or injection of fluids during the hydraulic fracturing process. Microseismic data, however, are often contaminated with strong random noise and have a low signal-to-noise ratio (S/N). The low S/N in most microseismic data severely affects the accuracy and reliability of the source localization and source-mechanism inversion results. We have developed a new denoising framework to enhance the quality of microseismic data. We use the method of adaptive sparse dictionaries to learn the waveform features of the microseismic data by iteratively updating the dictionary atoms and sparse coefficients in an unsupervised way. Unlike most existing dictionary learning applications in the seismic community, we learn the features from 1D microseismic data, thereby to learn 1D features of the waveforms. We develop a sparse dictionary learning framework and then prepare the training patches and implement the algorithm to obtain favorable denoising performance. We use extensive numerical examples and real microseismic data examples to demonstrate the validity of our method. Results show that the features of microseismic waveforms can be learned to distinguish signal patches and noise patches even from a single channel of microseismic data. However, more training data can make the learned features smoother and better at representing useful signal components.

A Knowledge Transfer Framework for Differentially Private Sparse Learning

We study the problem of estimating high dimensional models with underlying sparse structures while preserving the privacy of each training example. We develop a differentially private high-dimensional sparse learning framework using the idea of knowledge transfer. More specifically, we propose to distill the knowledge from a “teacher” estimator trained on a private dataset, by creating a new dataset from auxiliary features, and then train a differentially private “student” estimator using this new dataset. In addition, we establish the linear convergence rate as well as the utility guarantee for our proposed method. For sparse linear regression and sparse logistic regression, our method achieves improved utility guarantees compared with the best known results (Kifer, Smith and Thakurta 2012; Wang and Gu 2019). We further demonstrate the superiority of our framework through both synthetic and real-world data experiments.

Learning on Hypergraphs With Sparsity.

Hypergraph is a general way of representing high-order relations on a set of objects. It is a generalization of graph, in which only pairwise relations can be represented. It finds applications in various domains where relationships of more than two objects are observed. On a hypergraph, as a generalization of graph, one wishes to learn a smooth function with respect to its topology. A fundamental issue is to find suitable smoothness measures of functions on the nodes of a graph/hypergraph. We show a general framework that generalizes previously proposed smoothness measures and also generates new ones. To address the problem of irrelevant or noisy data, we wish to incorporate sparse learning framework into learning on hypergraphs. We propose sparsely smooth formulations that learn smooth functions and induce sparsity on hypergraphs at both hyperedge and node levels. We show their properties and sparse support recovery results. We conduct experiments to show that our sparsely smooth models are beneficial to learning irrelevant and noisy data, and usually give similar or improved performances compared to dense models.

Lung and Pancreatic Tumor Characterization in the Deep Learning Era: Novel Supervised and Unsupervised Learning Approaches.

Risk stratification (characterization) of tumors from radiology images can be more accurate and faster with computer-aided diagnosis (CAD) tools. Tumor characterization through such tools can also enable non-invasive cancer staging, prognosis, and foster personalized treatment planning as a part of precision medicine. In this papet, we propose both supervised and unsupervised machine learning strategies to improve tumor characterization. Our first approach is based on supervised learning for which we demonstrate significant gains with deep learning algorithms, particularly by utilizing a 3D convolutional neural network and transfer learning. Motivated by the radiologists' interpretations of the scans, we then show how to incorporate task-dependent feature representations into a CAD system via a graph-regularized sparse multi-task learning framework. In the second approach, we explore an unsupervised learning algorithm to address the limited availability of labeled training data, a common problem in medical imaging applications. Inspired by learning from label proportion approaches in computer vision, we propose to use proportion-support vector machine for characterizing tumors. We also seek the answer to the fundamental question about the goodness of "deep features" for unsupervised tumor classification. We evaluate our proposed supervised and unsupervised learning algorithms on two different tumor diagnosis challenges: lung and pancreas with 1018 CT and 171 MRI scans, respectively, and obtain the state-of-the-art sensitivity and specificity results in both problems.

Perceptually learning multi-view sparse representation for scene categorization

Utilizing multi-channel visual features to characterize scenery images is standard for state-of-the-art scene recognition systems. However, how to encode human visual perception for scenery image modeling and how to optimally combine visual features from multiple views remains a tough challenge. In this paper, we propose a perceptual multi-view sparse learning (PMSL) framework to distinguish sceneries from different categories. Specifically, we first project regions from each scenery into the so-called perceptual space, which is established by combining human gaze behavior, color and texture. Afterward, a novel PMSL is developed which fuzes the above three visual cues into a sparse representation. PMSL can support absent channel visual features, which is frequently occurred in practical circumstances. Finally, the sparse representation from each scenery image is incorporated into an image kernel, which is further fed into a kernel SVM for scene categorization. Comprehensive experimental results on popular data sets have demonstrated the superiority of our method over well-known shallow/deep recognition models.

Compressing by Learning in a Low-Rank and Sparse Decomposition Form

Low-rankness and sparsity are often used to guide the compression of convolutional neural networks (CNNs) separately. Since they capture global and local structure of a matrix respectively, we combine these two complementary properties together to pursue better network compression performance. Most existing low-rank or sparse compression methods compress the networks by approximating pre-trained models. However, the optimal solutions to pre-trained models may not be optimal to compressed networks with low-rank or sparse constraints. In this paper, we propose a low-rank and sparse learning framework that trains the compressed network from scratch. Our compressing process can be described as the following three stages. (a) In the structure designing stage, we decompose a weight matrix into sum of low-rank matrix and sparse matrix, and then the low-rank matrix is further factorized into product of two small matrices. (b) In training stage, we add l 1 regularization to the loss function to force the sparse matrix to be sparse. (c) In the post-processing stage, we remove the unimportant connection of sparse matrix according to its energy distribution. The pruning process in the post-processing stage reserves most of capacity of the network and keeps the performance of the network to a great extent. The performance can be further improved with fine-tuning, along with sparse masked convolution. Experiments on several common datasets demonstrate our model is superior to other network compression methods based on low-rankness or sparsity. On CIFAR-10, our method compresses VGGNet-19 to 3.14% and PreActResNet-56 to 29.78% without accuracy drop. 62.43% of parameters of ResNet-50 are reduced with 0.55% top-5 accuracy loss on ImageNet.

Fast Abnormal Event Detection

Fast abnormal event detection meets the growing demand to process an enormous number of surveillance videos. Based on the inherent redundancy of video structures, we propose an efficient sparse combination learning framework with both batch and online solvers. It achieves decent performance in the detection phase without compromising result quality. The extremely fast execution speed is guaranteed owing to the fact that our method effectively turns the original complicated problem into a few small-scale least square optimizations. Our method reaches high detection rates on benchmark datasets at a speed of 1000–1200 frames per second on average when computing on an ordinary single core desktop PC using MATLAB.