Sparse Representation Learning Research Articles

Towards Effective and Efficient Sparse Neural Information Retrieval

Sparse representation learning based on Pre-trained Language Models has seen a growing interest in Information Retrieval. Such approaches can take advantage of the proven efficiency of inverted indexes and inherit desirable IR priors such as explicit lexical matching or some degree of interpretability. In this work, we thoroughly develop the framework of sparse representation learning in IR, which unifies term weighting and expansion in a supervised setting. We then build on SPLADE—a sparse expansion-based retriever—and show to which extent it is able to benefit from the same training improvements as dense bi-encoders by studying the effect of distillation, hard negative mining, as well as the Pre-trained Language Model’s initialization on its effectiveness , leading to state-of-the-art results in both in- and out-of-domain evaluation settings (SPLADE++). We furthermore propose efficiency improvements, allowing us to reach latency requirements on par with traditional keyword-based approaches (Efficient-SPLADE).

Optimizing Image Recognition Efficiency using Sparse Representation Learning and Transfer Learning for Resource-Constrained Environments

Image recognition in resource-constrained environments is crucial for mobile devices, IoT, and embedded systems. It reduces the need for internet connectivity and enhances privacy and security. Real-time processing and decision-making are critical for applications such as facial recognition, object detection, and augmented reality. This approach is efficient, making it accessible in remote areas or situations with limited internet access. Our study proposes a novel approach using a network architecture with four modules, emphasizing sparse representation learning and transfer learning to improve image recognition efficiency in resource-constrained environments. The Domain-Adaptive Feature Extractor () facilitates effective sparse representation learning by projecting data from diverse domains into a shared space. The Transferable Affine Decoder () captures affine relationships between domains to facilitate knowledge transfer, while the Cross-Domain Correspondence Network () enforces pixel-level correspondence to extract shared intrinsic representations. The Efficient Classifier Network () enhances classification accuracy using efficient CNNs. The baseline model achieved an accuracy of 0.89. Improved Model 1, leveraging transfer learning, attained 0.92 accuracy, while Improved Model 2 with the Cross-Domain Correspondence Network reached 0.91. The Final Model, amalgamating all methodologies, excelled with the highest accuracy of 0.94. This holistic approach optimizes resource usage and enables real-time processing, thus empowering a diverse array of applications in resource-limited environments.

Blind passive signal detection via dictionary learning in unknown multipath time-spreading distortion underwater channels

Blind passive signal detection is a challenging endeavor since typically there is no prior knowledge of the transmitted signal. This becomes further complicated in time-spreading distortion (TSD) underwater channels [R. Rashid, E. Zhang, A. Abdi, and Z. H. Michalopoulou, “Theoretical and experimental multi-sensor signal detection in time spreading distortion underwater channels,” Proc. Oceans (2022)], where due to the presence of multiple propagation paths, the unknown signal is convolved with an unknown channel impulse response. In this paper, we introduce and develop a blind passive signal detection method, using dictionary learning [M. Sadeghi, M. Babaie-Zadeh, and C. Jutten, “Dictionary learning for sparse representation: A novel approach,” IEEE Signal Process. Lett. (2013)]. In our blind passive signal detection method, we use the received data to estimate the unknown signal, and also to separate it from the unknown channel impulse response. We have conducted simulations and underwater experiments, to generate receiver operating characteristic curves, to study the performance of the proposed method. The high detection probabilities of the blind method, compared to the replica correlation integration method—that needs to know the transmitted signal for matched filtering—demonstrate the usefulness of the method for passive detection of unknown signals in unknown TSD channels.

Augmented Sparse Representation for Incomplete Multiview Clustering.

Incomplete multiview data are collected from multiple sources or characterized by multiple modalities, where the features of some samples or some views may be missing. Incomplete multiview clustering (IMVC) aims to partition the data into different groups by taking full advantage of the complementary information from multiple incomplete views. Most existing methods based on matrix factorization or subspace learning attempt to recover the missing views or perform imputation of the missing features to improve clustering performance. However, this problem is intractable due to a lack of prior knowledge, e.g., label information or data distribution, especially when the missing views or features are completely damaged. In this article, we proposed an augmented sparse representation (ASR) method for IMVC. We first introduce a discriminative sparse representation learning (DSRL) model, which learns the sparse representations of multiple views as applied to measure the similarity of the existing features. The DSRL model explores complementary and consistent information by integrating the sparse regularization item and a consensus regularization item, respectively. Simultaneously, it learns a discriminative dictionary from the original samples. The sparsity constrained optimization problem in the DSRL model can be efficiently solved by the alternating direction method of multipliers (ADMM). Then, we present a similarity fusion scheme, namely, a sparsity augmented fusion of sparse representations, to obtain a sparsity augmented similarity matrix across different views for spectral clustering. Experimental results on several datasets demonstrate the effectiveness of the proposed ASR method for IMVC.

Kernel-Based Sparse Representation Learning With Global and Local Low-Rank Label Constraint

Kernel-Based Sparse Representation Learning With Global and Local Low-Rank Label Constraint

A tutorial-based survey on feature selection: Recent advancements on feature selection

Curse of dimensionality is known as big challenges in data mining, pattern recognition, computer vison and machine learning in recent years. Feature selection and feature extraction are two main approaches to circumvent this challenge. The main objective in feature selection is to remove the redundant features and preserve the relevant features in order to improve the learning algorithm performance. This survey provides a comprehensive overview of state-of-art feature selection techniques including mathematical formulas and fundamental algorithm to facilitate understanding. This survey encompasses different approaches of feature selection which can be categorized to five domains including: A) subspace learning which involves matrix factorization and matrix projection, B) sparse representation learning which includes compressed sensing and dictionary learning, C) information theory which covers multi-label neighborhood entropy, symmetrical uncertainty, Monte Carlo and Markov blanket, D) evolutionary computational algorithms including Genetic algorithm (GA), particle swarm optimization (PSO), Ant colony (AC) and Grey wolf optimization (GWO), and E) reinforcement learning techniques. This survey can be helpful for researchers to acquire deep understanding of feature selection techniques and choose a proper feature selection technique. Moreover, researcher can choose one of the A, B, C, D and E domains to become deep in this field for future study. A potential avenue for future research could involve exploring methods to reduce computational complexity while simultaneously maintaining performance efficiency. This would involve investigating ways to achieve a more efficient balance between computational resources and overall performance. For matrix-based techniques, the main limitation of these techniques lies in the need to tune the coefficients of the regularization terms, as this process can be challenging and time-consuming. For evolutionary computational techniques, getting stuck in local minimum and finding an appropriate objective function are two main limitations.

Efficient Sparse Representation for Learning With High-Dimensional Data.

Due to the capability of effectively learning intrinsic structures from high-dimensional data, techniques based on sparse representation have begun to display an impressive impact on several fields, such as image processing, computer vision, and pattern recognition. Learning sparse representations isoften computationally expensive due to the iterative computations needed to solve convex optimization problems in which the number of iterations is unknown before convergence. Moreover, most sparse representation algorithms focus only on determining the final sparse representation results and ignore the changes in the sparsity ratio (SR) during iterative computations. In this article, two algorithms are proposed to learn sparse representations based on locality-constrained linear representation learning with probabilistic simplex constraints. Specifically, the first algorithm, called approximated local linear representation (ALLR), obtains a closed-form solution from individual locality-constrained sparse representations. The second algorithm, called ALLR with symmetric constraints (ALLRSC), further obtains a symmetric sparse representation result with a limited number of computations; notably, the sparsity and convergence of sparse representations can be guaranteed based on theoretical analysis. The steady decline in the SR during iterative computations is a critical factor in practical applications. Experimental results based on public datasets demonstrate that the proposed algorithms perform better than several state-of-the-art algorithms for learning with high-dimensional data.

Sparse representation learning using [formula omitted] compressed sensing and rank-revealing QR factorization

Compressive sensing can be conceptualized for a classification problem in the context of representation learning, which is frequently applied for signal reconstruction via a few measurements. A novel compressive sensing-based approach is presented to substantially reduce the number of required samples for classification. This technique considers the pixels of image as sensors and it seeks to identify the most optimum sensors as decision variables in feature space. In this study, Spatial sensor locations are acquired through learning to identify the regions within an image where the most discriminative information for classification is embedded. ℓ1−2 minimization, as nonconvex but Lipschitz continuous, is solved to obtain the least non-zero elements of the full measurement vector to completely reconstruct the discriminative vector in feature space. Optimal sensors are localized from the training-set and subsequent test images are categorized based on learned sensors. The algorithm consists of three primary components: sparse minimization, feature space, and discrimination vector. ℓ1−2 minimization, rank-revealing QR factorization (RRQR) and SVM are considered for enhanced sparsity exploitation, feature space extraction and discrimination vector acquiring, respectively. The proposed method is evaluated on four different experiments and is compared against a state-of-art technique. The results demonstrate the superiority of proposed method to the compared method.

Sparse representation learning for fault feature extraction and diagnosis of rotating machinery

Early fault feature extraction and fault diagnosis are of great importance for predictive maintenance of rotating machinery. To accurately extract early fault features from original noisy signals, a novel joint sparse representation learning method is developed in this paper, this method is based on the proposed nonlocal generalized minimax-concave (GMC) penalty and generalized fraction-order total variation (FTV) regularization. The motivation for this research is to leverage the benefits of joint regularizations. The proposed nonlocal GMC penalty regularization tends to preserve weak fault features, promote sparsity and avoid underestimating the amplitude of periodic fault impulses. Simultaneously, the proposed generalized FTV regularization tends to remove fault irrelevant noise and reduce staircase artifacts. Therefore, the proposed model can effectively extract early fault features from original noisy signals. The performance of the proposed model is verified by a series of experiments. In two fault diagnosis tasks, the peak signal-to-noise ratio (PSNR) of the proposed method reaches − 5 dB and − 8 dB, respectively. Compared with state-of-the-art methods, the PSNR has been improved by at least 2 dB, comparison results show that the proposed model has superior performance for early fault feature extraction.

Sparse representation learning with modified q-VAE towards minimal realization of world model

Extraction of low-dimensional latent space from high-dimensional observation data is essential to construct a real-time robot controller with a world model on the extracted latent space. However, there is no established method for tuning the dimension size of the latent space automatically, suffering from finding the necessary and sufficient dimension size, i.e. the minimal realization of the world model. In this study, we analyze and improve Tsallis-based variational autoencoder (q-VAE), and reveal that, under an appropriate configuration, it always facilitates making the latent space sparse. Even if the dimension size of the pre-specified latent space is redundant compared to the minimal realization, this sparsification collapses unnecessary dimensions, allowing for easy removal of them. We experimentally verified the benefits of the sparsification by the proposed method that it can easily find the necessary and sufficient six dimensions for a reaching task with a mobile manipulator that requires a six-dimensional state space. Moreover, by planning with such a minimal-realization world model learned in the extracted dimensions, the proposed method was able to exert a more optimal action sequence in real-time, reducing the reaching accomplishment time by around 20%.

Sparse representation learning derives biological features with explicit gene weights from the Allen Mouse Brain Atlas.

Unsupervised learning methods are commonly used to detect features within transcriptomic data and ultimately derive meaningful representations of biology. Contributions of individual genes to any feature however becomes convolved with each learning step, requiring follow up analysis and validation to understand what biology might be represented by a cluster on a low dimensional plot. We sought learning methods that could preserve the gene information of detected features, using the spatial transcriptomic data and anatomical labels of the Allen Mouse Brain Atlas as a test dataset with verifiable ground truth. We established metrics for accurate representation of molecular anatomy to find sparse learning approaches were uniquely capable of generating anatomical representations and gene weights in a single learning step. Fit to labeled anatomy was highly correlated with intrinsic properties of the data, offering a means to optimize parameters without established ground truth. Once representations were derived, complementary gene lists could be further compressed to generate a low complexity dataset, or to probe for individual features with >95% accuracy. We demonstrate the utility of sparse learning as a means to derive biologically meaningful representations from transcriptomic data and reduce the complexity of large datasets while preserving intelligible gene information throughout the analysis.

Efficient Document-at-a-Time and Score-at-a-Time Query Evaluation for Learned Sparse Representations

Researchers have had much recent success with ranking models based on so-called learned sparse representations generated by transformers. One crucial advantage of this approach is that such models can exploit inverted indexes for top- k retrieval, thereby leveraging decades of work on efficient query evaluation. Yet, there remain many open questions about how these learned representations fit within the existing literature, which our work aims to tackle using four representative learned sparse models. We find that impact weights generated by transformers appear to greatly reduce opportunities for skipping and early exiting optimizations in well-studied document-at-a-time ( DaaT ) approaches. Similarly, “off-the-shelf” application of score-at-a-time ( SaaT ) processing exhibits a mismatch between these weights and assumptions behind accumulator management strategies. Building on these observations, we present solutions to address deficiencies with both DaaT and SaaT approaches, yielding substantial speedups in query evaluation. Our detailed empirical analysis demonstrates that both methods lie on the effectiveness–efficiency Pareto frontier, indicating that the optimal choice for deployment depends on operational constraints.

MADPL-net: Multi-layer attention dictionary pair learning network for image classification

With the great success of deep neural networks, combining deep learning with traditional dictionary learning has become a hot issue. However, the performance of these methods is still limited for several reasons. First, some existing methods update dictionary learning and classifier as two independent modules, which limits the classification performance. Second, the non-attention dictionary is learned to represent all images, reducing the model representation flexibility. In this paper, we design a novel end-to-end model named Multi-layer Attention Dictionary Pair Learning Network (MADPL-net), which integrates the learning schemes of the convolutional neural network, deep encoder learning, and attention dictionary pair learning (ADicL) into a unified framework. The encoder layer contains the ADicL block, which selects more image-attentive atoms in the dictionary pair block via the softmax function to ensure MADPL-net classification capability. In addition, ADicL schema can yield discriminative dictionary atoms and feature maps with high inter-class separation and high intra-class compactness. To improve the sparse representation learning performance, MADPL-net adds l1−norm constraint of the analysis dictionary to the cross-entropy loss function. Extensive experiments show that MADPL-net can achieve excellent performance over other state-of-the-arts.

Chemical mixture exposure patterns and obesity among U.S. adults in NHANES 2005–2012

BackgroundThe effect of chemical exposure on obesity has raised great concerns. Real-world chemical exposure always imposes mixture impacts, however their exposure patterns and the corresponding associations with obesity have not been fully evaluated. ObjectivesTo discover obesity-related mixed chemical exposure patterns in the general U.S. population. MethodsSparse Decompositional Regression (SDR), a model adapted from sparse representation learning technique, was developed to identify exposure patterns of chemical mixtures with exclusion (non-targeted model) and inclusion (targeted model) of health outcomes. We assessed the relationships between the identified chemical mixture patterns and obesity-related indexes. We also conducted a comprehensive evaluation of this SDR model by comparing to the existing models, including generalized linear regression model (GLM), principal component analysis (PCA), and Bayesian kernel machine regression (BKMR). ResultsEight core exposure patterns were identified using the non-targeted SDR model. Patterns of high levels of MEP, high levels of naphthalene metabolites (ΣOH-Nap), and a pattern of high exposure levels of MCOP, MCNP, and MCPP were positively associated with obesity. Patterns of high levels of BP3, and a pattern of higher mixed levels of MPB, PPB, and MEP were found to have negative associations. Associations were strengthened using the targeted SDR model. In the single chemical analysis by GLM, BP3, MBP, PPB, MCOP, and MCNP showed significant associations with obesity or body indexes. The SDR model exceeded the performance of PCA in pattern identification. Both SDR and BKMR identified a positive contribution of ΣOH-Nap and MCOP, as well as a negative contribution of BP3 and PPB to obesity. ConclusionOur study identified five core exposure patterns of chemical mixtures significantly associated with obesity using the newly developed SDR model. The SDR model could open a new avenue for assessing health effects of environmental mixture contaminants.

ICEMIGE: Integration of CEll-morphometrics, MIcrobiome, and GEne biomarker signatures for risk stratification in breast cancers.

BACKGROUNDThe development of precision medicine is essential for personalized treatment and improved clinical outcome, whereas biomarkers are critical for the success of precision therapies. AIMTo investigate whether iCEMIGE (integration of CEll-morphometrics, MIcro biome, and GEne biomarker signatures) improves risk stratification of breast cancer (BC) patients.METHODSWe used our recently developed machine learning technique to identify cellular morphometric biomarkers (CMBs) from the whole histological slide images in The Cancer Genome Atlas (TCGA) breast cancer (TCGA-BRCA) cohort. Multivariate Cox regression was used to assess whether cell-morphometrics prognosis score (CMPS) and our previously reported 12-gene expression prognosis score (GEPS) and 15-microbe abundance prognosis score (MAPS) were independent prognostic factors. iCEMIGE was built upon the sparse representation learning technique. The iCEMIGE scoring model performance was measured by the area under the receiver operating characteristic curve compared to CMPS, GEPS, or MAPS alone. Nomogram models were created to predict overall survival (OS) and progress-free survival (PFS) rates at 5- and 10-year in the TCGA-BRCA cohort.RESULTSWe identified 39 CMBs that were used to create a CMPS system in BCs. CMPS, GEPS, and MAPS were found to be significantly independently associated with OS. We then established an iCEMIGE scoring system for risk stratification of BC patients. The iGEMIGE score has a significant prognostic value for OS and PFS independent of clinical factors (age, stage, and estrogen and progesterone receptor status) and PAM50-based molecular subtype. Importantly, the iCEMIGE score significantly increased the power to predict OS and PFS compared to CMPS, GEPS, or MAPS alone. CONCLUSIONOur study demonstrates a novel and generic artificial intelligence framework for multimodal data integration toward improving prognosis risk stratification of BC patients, which can be extended to other types of cancer.

DCSRL: a change detection method for remote sensing images based on deep coupled sparse representation learning

ABSTRACT Change detection methods include model-driven methods and data-driven methods. Model-driven methods are rigorous in theory but have difficulties in establishing accurate models for complex problems. Data-driven methods have strong learning abilities, but it is difficult to interpret their mechanism. In order to comprehensively utilize their advantages to highlight changes in multi-temporal remote sensing images. We proposed a change detection method for remote sensing images based on deep coupled sparse representation learning, in which the coupled sparse representation learning is implicitly expressed by convolutional neural networks (CNNs). A coupled sparse representation learning method is first proposed, and then it is unfolded into a CNNs network, the coupled sparse coefficient and dictionary are learned from the training data. Unlike fully data-driven CNNs, auxiliary coupled sparse representation is utilized to guide CNNs to identify the changed areas. Finally, the experiments on two datasets verified the validity of the proposed method.

Dictionary learning for sparse representation of signals with hidden Markov model dependency

The goal of dictionary learning algorithms is factorizing the matrix of training signals Y with K signals, into the dictionary matrix D and the coefficient matrix X which is a sparse matrix. The common approach among the algorithms is minimizing the representation error subject to the sparseness of X using alternation minimization method following 1) the sparsification and 2) the dictionary update steps. In this approach, when D is fixed and X must be estimated (the sparsification step), the minimization problem is divided to K different sparse recovery problems because the training signals are assumed to be independent. However, in some signals this assumption is not correct and the signals are not independent. Therefore, using the current strategy does not lead to the correct estimation of the parameters. The prominence is especially linked to medical signals and images such as electroencephalography (EEG) or diffusion weighted images, where the recordings do not constitute a matrix of independent training signals. In this study, we investigate the dictionary learning problem for sparse representation when there is hidden Markov model (HMM) dependency among the training signals. We propose an approach to improve the performance of the dictionary learning algorithms in the mentioned scenario. The proposed approach is not an independent dictionary learning algorithm. It is a general approach that we can employ for existing dictionary learning algorithms to improve their performance in learning from signals with HMM dependency. We confirm the efficiency of the proposed approach using simulations, and also, present a real application in medical signals for the considered scenario.

Image Compression Using Stochastic-AFD Based Multisignal Sparse Representation.

Adaptive Fourier decomposition (AFD) is a newly developed signal processing tool that can adaptively decompose any single signal using a Szegö kernel dictionary. To process multiple signals, a novel stochastic-AFD (SAFD) theory was recently proposed. The innovation of this study is twofold. First, a SAFD-based general multi-signal sparse representation learning algorithm is designed and implemented for the first time in the literature, which can be used in many signal and image processing areas. Second, a novel SAFD based image compression framework is proposed. The algorithm design and implementation of the SAFD theory and image compression methods are presented in detail. The proposed compression methods are compared with 13 other state-of-the-art compression methods, including JPEG, JPEG2000, BPG, and other popular deep learning-based methods. The experimental results show that our methods achieve the best balanced performance. The proposed methods are based on single image adaptive sparse representation learning, and they require no pre-training. In addition, the decompression quality or compression efficiency can be easily adjusted by a single parameter, that is, the decomposition level. Our method is supported by a solid mathematical foundation, which has the potential to become a new core technology in image compression.

Image Classification Based on Sparse Representation in the Quaternion Wavelet Domain

In this study, we propose a novel sparse representation learning method in the Quaternion Wavelet (QW) domain for multi-class image classification. The proposed method takes advantages from: i) the QW decomposition, which promotes sparsity and provides structural information about the image data while allowing approximate shift-invariance, to extract meaningful features from low-frequency QW sub-bands, ii) the dimensionality reduction method using Principal Component Analysis (PCA) for reducing the complexity of the problem, and iii) the sparse representation of the generated QW features to efficiently learn and capture the meaningful and compact information of this data. After the QW decomposition, the features extracted from low-frequency image sub-bands information are projected, by the PCA, into a new feature space with lower dimensionality. The features extracted from the training samples are used to construct a dictionary, while the features of the test samples are sparsely coded for the classification step. The sparse coding problem is formulated in a QW Least Absolute Shrinkage and Selection Operator (QWLasso) model applying quaternion <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$l_{1}$ </tex-math></inline-formula> minimization. A novel Quaternion Fast Iterative Shrinkage-Thresholding Algorithm (QFISTA) is developed to solve the QWLasso model. The experiments conducted on various public image datasets validated that the proposed method possesses higher accuracy, sparsity, and robustness in comparison with several contemporary methods in the field including Neural Networks.

Robust total variation-based destriping model via sparse representation learning for business infrared imaging systems

Infrared images are often degraded by the issue of random and stripe noises in the business infrared focal plane imaging process. In this study, we proposed a novel infrared image destriping model with unidirectional and bidirectional total variation regularization. From the destriped infrared images, the stripe noise only exists in unidirection. The nature of unidirectional noise is leveraged to converted the total variation regularization and constructed the proposed model. For the random noise, the bidirectional total variation is introduced, which can preserve the image edge and texture details. Furthermore, the alternating direction method of multiplier algorithm is introduced to solve the infrared image destriping model. The proposed model is executed on several public infrared image datasets, thus outperforming the traditional destriping models in the qualitative evaluation index.