Sparse Model Research Articles

Line segment detectors with deformable attention

The object detectors based on Transformers are advancing rapidly. On the contrary, the development of line segment detectors is relatively slow. It is noteworthy that the object and line segments are both 2D targets. In this work, we design a line segment detection algorithm based on deformable attention. Leveraging this algorithm and the line segment loss function, we transform the object detectors, Deformable DETR and ViDT, into end-to-end line segment detectors named Deformable LETR and ViDTLE, respectively. In order to adapt the idea of sparse modeling for line segment detection, we propose a new attention mechanism named line segment deformable attention (LSDA). This mechanism focuses on the valuable positions under the guidance of reference line to refine line segments. We design an auxiliary algorithm named line segment iterative refinement for LSDA. With as few modifications as possible, we transform two object detection detectors, namely SMCA DETR and PnP DETR into competitive line segment detectors named SMCA LETR and PnP LETR, respectively. The experimental results show that the performances of the proposed methods are efficient.

Scalp surface estimation and head registration using sparse sampling and 3D statistical models

Registering the head and estimating the scalp surface are important for various biomedical procedures, including those using neuronavigation to localize brain stimulation or recording. However, neuronavigation systems rely on manually-identified fiducial head targets and often require a patient-specific MRI for accurate registration, limiting adoption. We propose a practical technique capable of inferring the scalp shape and use it to accurately register the subject’s head. Our method does not require anatomical landmark annotation or an individual MRI scan, yet achieves accurate registration of the subject’s head and estimation of its surface. The scalp shape is estimated from surface samples easily acquired using existing pointer tools, and registration exploits statistical head model priors. Our method allows for the acquisition of non-trivial shapes from a limited number of data points while leveraging their object class priors, surpassing the accuracy of common reconstruction and registration methods using the same tools. The proposed approach is evaluated in a virtual study with head MRI data from 1152 subjects, achieving an average reconstruction root-mean-square error of 2.95 mm, which outperforms a common neuronavigation technique by 2.70 mm. We also characterize the error under different conditions and provide guidelines for efficient sampling. Furthermore, we demonstrate and validate the proposed method on data from 50 subjects collected with conventional neuronavigation tools and setup, obtaining an average root-mean-square error of 2.89 mm; adding landmark-based registration improves this error to 2.63 mm. The simulation and experimental results support the proposed method’s effectiveness with or without landmark annotation, highlighting its broad applicability.

Simultaneous clustering and estimation of networks in multiple graphical models.

Gaussian graphical models are widely used to study the dependence structure among variables. When samples are obtained from multiple conditions or populations, joint analysis of multiple graphical models are desired due to their capacity to borrow strength across populations. Nonetheless, existing methods often overlook the varying levels of similarity between populations, leading to unsatisfactory results. Moreover, in many applications, learning the population-level clustering structure itself is of particular interest. In this article, we develop a novel method, called Simultaneous Clustering and Estimation of Networks via Tensor decomposition (SCENT), that simultaneously clusters and estimates graphical models from multiple populations. Precision matrices from different populations are uniquely organized as a three-way tensor array, and a low-rank sparse model is proposed for joint population clustering and network estimation. We develop a penalized likelihood method and an augmented Lagrangian algorithm for model fitting. We also establish the clustering accuracy and norm consistency of the estimated precision matrices. We demonstrate the efficacy of the proposed method with comprehensive simulation studies. The application to the Genotype-Tissue Expression multi-tissue gene expression data provides important insights into tissue clustering and gene coexpression patterns in multiple brain tissues.

Comparing statistical learning methods for complex trait prediction from gene expression.

Accurate prediction of complex traits is an important task in quantitative genetics that has become increasingly relevant for personalized medicine. Genotypes have traditionally been used for trait prediction using a variety of methods such as mixed models, Bayesian methods, penalized regressions, dimension reductions, and machine learning methods. Recent studies have shown that gene expression levels can produce higher prediction accuracy than genotypes. However, only a few prediction methods were used in these studies. Thus, a comprehensive assessment of methods is needed to fully evaluate the potential of gene expression as a predictor of complex trait phenotypes. Here, we used data from the Drosophila Genetic Reference Panel (DGRP) to compare the ability of several existing statistical learning methods to predict starvation resistance from gene expression in the two sexes separately. The methods considered differ in assumptions about the distribution of gene effect sizes - ranging from models that assume that every gene affects the trait to more sparse models - and their ability to capture gene-gene interactions. We also used functional annotation (i.e., Gene Ontology (GO)) as an external source of biological information to inform prediction models. The results show that differences in prediction accuracy between methods exist, although they are generally not large. Methods performing variable selection gave higher accuracy in females while methods assuming a more polygenic architecture performed better in males. Incorporating GO annotations further improved prediction accuracy for a few GO terms of biological significance. Biological significance extended to the genes underlying highly predictive GO terms with different genes emerging between sexes. Notably, the Insulin-like Receptor (InR) was prevalent across methods and sexes. Our results confirmed the potential of transcriptomic prediction and highlighted the importance of selecting appropriate methods and strategies in order to achieve accurate predictions.

SAR target recognition through adaptive kernel sparse representation model based on local contrast perception

Sparse representation-based methods have shown promising prospects in synthetic aperture radar (SAR) target recognition due to their robustness under complex application conditions. However, these methods are still susceptible to the feature representation of SAR images. In this paper, we propose a new SAR target recognition method using an adaptive kernel sparse representation model based on local contrast perception (LCP). The LCP method combines a specially designed random projection matrix with two-dimensional filtering to capture extensive intensity contrasts between local regions in SAR images. A response map reconstruction technique shows that LCP can obtain discriminative properties between target and background regions and improve feature representation. The LCP features are then transformed into the kernel representation to improve linear separability by defining a reproducing kernel Hilbert space (RKHS) and using a nonlinear kernel function. The kernel representation is utilized to construct a new kernel sparse representation model for classification. Furthermore, neighborhood analysis (NA) is proposed to establish the relationship between the parameter and feature distribution, with an adaptive regularization parameter selection strategy to mitigate the parameter sensitivity. Experiments conducted on the moving and stationary target acquisition and recognition dataset demonstrate the superiority and robustness of the proposed method under different application conditions.

High Performance CNN Accelerators based on Hardware and Algorithm Co-Optimization

The efficacy of convolutional neural networks (CNNs) has led to their extensive application in picture recognition and categorization. Nevertheless, embedded systems face challenges when it comes to storing the massive volumes of weight data required by CNNs in their on-chip memory. Pruning a convolutional neural network (CNN) model can reduce its size with no impact on accuracy; however, when used with a parallel architecture, the resulting model will be slower to execute. A CNN compression method that is hardware-centric is introduced in this paper. A model of a deep neural network (DNN) will include "pruning layers (P-layers)" and "no-pruning layers (NP-layers)" attached to it. For effective and parallel processing, an NP-layer employs a regular distribution for its weights. Pruning causes a P-layer to have an irregular form, yet the high compression ratio it yields makes the shape undesirable. Using uniform and gradual quantization methods, we can achieve a processing efficiency/compression ratio trade-off with a small loss of precision. Integrating multiple parallel finite impulse response (FIR) filters into an NP-layer distributed convolutional architecture, we further improve the regular model. The P-layer irregular sparse model is suggested to use an ADF-based processing element. The suggested compression strategy and hardware architecture can be utilised by a hardware/algorithm co-optimization (HACO) method to run an NP→P hybrid compressed CNN model on FPGAs. The VGG-16 network achieves a compression ratio of 27.5× with a top-5 accuracy loss of 0.44% by using a hardware accelerator on a single FPGA chip without off-chip memory. A result of 83.0 FPS, or 1.8 times quicker, was achieved when the compressed VGG-16 model was implemented on a Xilinx VCU118 evaluation board for image applications. Index Terms - CNN, Accelerators, Co-Optimization.

Channel Adaptive and Sparsity Personalized Federated Learning for Privacy Protection in Smart Healthcare Systems.

With the booming development of Smart Healthcare Systems (SHSs), employing federated learning (FL) in SHS devices has become a research hotspot. FL, as a distributed learning framework, can train models without sharing the original data among users, and then protect the user privacy. Existing research has proposed many methods to improve the security and efficiency of FL, which may not fully consider the characteristics of SHSs. Specifically, the requirements of privacy protection and efficiency pose significant challenges to FL. Current studies have struggled to balance privacy security and efficiency, and the degradation of model training efficiency in SHSs can be critical to patient health. Therefore, to improve the privacy protection of healthcare data and ensure communication efficiency, this work proposes a novel personalized FL framework based on Communication quality and Adaptive Sparsification (pFedCAS). In order to achieve privacy protection, a control unit is proposed and introduced to adjust the sparsity of the local model adaptively. To further improve the training efficiency, a selection unit is added during global model aggregation to select suitable clients for parameter updates. Finally, we validate the proposed method operated on the HAM10000 dataset. Simulation results validate that pFedCAS can not only improve privacy protection, but also gain an improvement of 15% in training accuracy and a reduction of 30% in training costs based on communication quality. The simulation results also validate the excellent robustness of pFedCAS to non-iid data.

Time analysis for aero-engine acoustic modes exploiting block sparsity

Acoustic Mode Analysis (AMA) for aero-engines can offer valuable insights for the design of silent engines as well as for fault diagnosis. Commonly, this is done in the (spatial) Fourier domain, necessitating the use of multiple uniformly spaced microphones to ensure adequate resolution. Recent works show that sub-Nyquist estimation is feasible using sparse reconstruction frameworks, although such modelling generally introduces an estimation bias that has to be compensated for. Moreover, there is a growing interest in monitoring mode amplitude over continuous time, as it can offer crucial insights for diagnosing operational conditions. In this work, we introduce a Block Orthogonal Matching Pursuit (BOMP) method for continuous time mode analysis, exploiting the underlying structural sparsity of the signal model. Specifically, the (pseudo) ℓ0−norm penalty is employed to induce sparsity in the wavenumber domain, whereas a block structure is imposed as a constraint to monitor the amplitude variation in the time domain. The effectiveness of the BOMP is evaluated using both numerical simulations and experimental measurements, indicating the proposed method’s preferable performance as compared to the classic Least Absolute Shrinkage and Selection Operator (LASSO) and Orthogonal Matching Pursuit (OMP) methods.

Typical structure of sparse exponential random graph models

Typical structure of sparse exponential random graph models

Sparse multi-view image clustering with complete similarity information

Multi-view image clustering aims to efficiently divide the collection of images by studying the characteristics of different views. Many studies performed Laplacian dimensionality reduction on the original image to avoid noise interference in constructing the similarity matrix. However, they ignored the problem that local similarity information and isolated image information are lost due to dimensionality reduction. These problems will result in the similarity matrix being insufficient to accurately describe the similarity between images, which in turn affects the clustering accuracy. Therefore, we propose a sparse multi-view spectral clustering model with complete similarity information between images (SMSC-CSI). The model combines the original image space and the low-dimensional spectral embedding space based on the adaptive neighbors method to learn the initial similarity matrices of each view jointly. On the one hand, the original space can retain all the similarity information between images so that the initial similarity matrix can accurately describe the similarity between images, which is conducive to accurate clustering. On the other hand, the low-dimensional space can avoid noise interference and retain the main structure of high-dimensional data, improving the robustness of the initial similarity matrix. Meanwhile, to ensure consistency among the views, the model minimizes the difference between the initial similarity matrix and the central fusion matrix by alternately updating to obtain the optimal weights and central fusion matrix for each view. Finally, we can obtain ideal clustering results directly with low model complexity and without post-processing steps such as K-means by adding non-negative Laplace rank constraints and ℓ0-norm constraints to the model objective function. Experimental results on different real image datasets show that SMSC-CSI can outperform some traditional clustering models and recent models on multi-view image clustering.

Enhancing model identification with SINDy via nullcline reconstruction.

Many dynamical systems exhibit oscillatory behavior that can be modeled with differential equations. Recently, these equations have increasingly been derived through data-driven methods, including the transparent technique known as Sparse Identification of Nonlinear Dynamics (SINDy). This paper illustrates the importance of accurately determining the system's limit cycle position in phase space for identifying sparse and effective models. We introduce a method for identifying the limit cycle position and the system's nullclines by applying SINDy to datasets adjusted with various offsets. This approach is evaluated using three criteria: model complexity, coefficient of determination, and generalization error. We applied this method to several models: the oscillatory FitzHugh-Nagumo model, a more complex model consisting of two coupled cubic differential equations with a single stable state, and a multistable model of glycolytic oscillations. Our results confirm that incorporating detailed information about the limit cycle in phase space enhances the accuracy of model identification in oscillatory systems.

Robust Integrative Analysis via Quantile Regression with Homogeneity and Sparsity

Integrative analysis plays a critical role in integrating heterogeneous data from multiple datasets to provide a comprehensive view of the overall data features. However, in multiple datasets, outliers and heavy-tailed data can render least squares estimation unreliable. In response, we propose a Robust Integrative Analysis via Quantile Regression (RIAQ) that accounts for homogeneity and sparsity in multiple datasets. The RIAQ approach is not only able to identify latent homogeneous coefficient structures but also recover the sparsity of high-dimensional covariates via double penalty terms. The integration of sample information across multiple datasets improves estimation efficiency, while a sparse model improves model interpretability. Furthermore, quantile regression allows the detection of subgroup structures under different quantile levels, providing a comprehensive picture of the relationship between response and high-dimensional covariates. We develop an efficient alternating direction method of multipliers (ADMM) algorithm to solve the optimization problem and study its convergence. We also derive the parameter selection consistency of the modified Bayesian information criterion. Numerical studies demonstrate that our proposed estimator has satisfactory finite-sample performance, especially in heavy-tailed cases.

АНАЛІЗ МЕТОДІВ ЗНАХОДЖЕННЯ АНОМАЛІЙ У ЗОБРАЖЕННЯХ

Abstract. Automatic anomaly detection is of great importance in industry, remote sensing, and medicine. It is important to be able to automatically process large amounts of data to detect, for example, chemical objects in multispectral and hyperspectral satellite images, sea mines in side-scan sonar images, or defects in industrial monitoring applications. Automatic detection of anomalous structures on arbitrary images refers to the task of finding inappropriate patterns relative to the normal state of the image. This is a difficult task in computer vision, since there is no clear and straightforward definition of what is normal or not normal for a given arbitrary image. The practical importance is manifested in the development of algorithms and models that can automatically detect unusual or anomalous patterns in images. An analysis of methods for finding anomalies in images from the point of view of the possibility of application to arbitrary images has been carried out. The classification of anomaly detection methods according to the criteria of the involved approaches and models used for modeling the background is presented. Methods that use machine learning, such as one-class support vector method and variational autoencoder, nearest neighbor-based anomaly detection, clustering-based anomaly detection, statistical anomaly detection, spectral anomaly detection, anomaly detection using information theory are discussed. The main attention is paid to the methods classified according to the background modeling approach. Five categories of background modeling methods based on probability density function, global and local homogeneity, sparsity, and self-similarity are considered. For anomaly detection applications, it is recommended to use methods in which the background model best describes the expected anomaly-free background, as this generally results in the best performance. On the basis of research, it was established that an effective universal model for detecting anomalies in arbitrary images should: use only a self-similar or sparse background model; process the residual image as a stochastic process to detect anomalies, such as anomalies in color noise; preprocess the residual image before detecting the anomaly.

Unveiling pathogenesis and therapeutic targets of metastatic breast cancer through sparse modeling of genomic landscape

Breast cancer is one of the most lethal and frequently detected diseases in the world. Its heterogeneity poses significant challenges for precision therapy. To better decipher patterns in the human genome given its heterogenous nature and have them converge into common functionalities, mutational signatures are introduced to define the types of DNA damage, repair, and replicative mechanisms shaping the genomic landscape of each cancer patient. In this study, we developed a deep learning (DL) model, based on pruning technology to improve model generalization through deep sparsity. We applied it to patient-level whole genome sequencing (WGS) samples, and identified statistically significant mutational signatures associated with metastatic progression using Shapley additive explanations (SHAP). We also employed gene cumulative contribution abundance analysis to link the mutational signatures with relevant genes uncovering the shared molecular mechanisms behind tumorigenesis and metastasis of each patient which can potentially lead to novel therapeutic target identification. Our study illustrates that our approach is an effective tool for discovering clinically meaningful mutational signatures in metastatic breast cancer (MBC) and relating them directly to relevant biological functions and gene targets. These findings could facilitate the development of novel therapeutic strategies and improve the clinical outcomes for individual patients.

Prediction of nanocomposite properties and process optimization using persistent homology and machine learning

Physical property prediction and synthesis process optimization are key targets in material informatics. In this study, we propose a machine learning approach that utilizes ridge regression to predict the oxygen permeability at fuel cell electrode surfaces and determine the optimal process temperature. These predictions are based on a persistence diagram derived from tomographic images captured using transmission electron microscopy (TEM). Through machine learning analysis of the complex structures present in the Pt/CeO2 nanocomposites, we discovered that l2 regularization considering diverse structural elements is more appropriate than l1 regularization (sparse modeling). Notably, our model successfully captured the activation energy of oxygen permeability, a phenomenon that could not be solely explained by the geometric feature of the Betti numbers, as demonstrated in a previous study. The correspondence between the ridge regression coefficient and persistence diagram revealed the formation process of the local and three-dimensional structures of CeO2 and their contributions to pre-exponential factor and activation energies. This analysis facilitated the determination of the annealing temperature required to achieve the optimal structure and accurately predict the physical properties.

Bayesian quantile inference and order shrinkage for hysteretic quantile autoregressive models

Hysteretic quantile autoregressive model combines the hysteretic patterns and quantile autoregression, which can capture the dynamic relationship and nonlinear characteristics at different quantiles in time series data. In this paper, the Bayesian quantile inference and order shrinkage are studied for a class of hysteretic quantile autoregressive time series models. By using Markov Chain Monte Carlo (MCMC) techniques, the proposed Bayesian quantile method can handle the sparse hysteretic quantile autoregressive model well. It can accurately determine order of the model and estimate non-zero coefficients. Both simulation studies and a data example show that the proposed methods are feasible, reliable and appropriate for analysing the US Gross National Product data set.

Optimized COVID-19 detection using sparse deep learning models from multimodal imaging data

Optimized COVID-19 detection using sparse deep learning models from multimodal imaging data

Active Learning of Ensemble Polynomial Chaos Expansion Method for Global Sensitivity Analysis

Global sensitivity analysis (GSA) techniques, especially Sobol indices, have gained prominence in assessing the impact of uncertainty across all input variables on the variation of model response. However, GSA can incur intensive computational costs, particularly when dealing with time-consuming models. To address this issue, an ensemble polynomial chaos expansion (EPCE) surrogate model based on an active learning hybrid (ALH) criterion is proposed. The proposed method integrates a weighted combination of multiple sparse PCE surrogate models to formulate the EPCE method. To enhance the performance, the ALH criterion, which balances local exploitation and global exploration, is developed to sequentially select samples for updating the EPCE model. In terms of local exploitation, the model correlation and sensitivity pursuit criteria are derived to identify the local region displaying both the largest discrepancy among multiple sparse PCE methods and the most substantial contribution to the total variance. The ALH criterion also incorporates a minimum distance-based measure for global exploration. The performance of the proposed method is evaluated by four benchmark functions and an engineering application. The numerical studies demonstrate that the proposed method exhibits favorable performance for conducting GSA.

Intelligent Diagnosis of Compound Faults of Gearboxes Based on Periodical Group Sparse Model

A gearbox compound fault intelligent diagnosis method based on the period group sparse model is proposed for the problem that the fault features are coupled with each other and the fault components are superimposed on each other and difficult to be separated in the gearbox compound fault signal. Firstly, a binary sequence is constructed to embed the fault pulse period as a priori knowledge into the group sparse model to decouple and separate the composite fault signal while maintaining the amplitude and sparsity of the extracted features. Secondly, the wavelet packet energy features of the decoupled signals are extracted to improve the data quality while enhancing the characterization ability of the dictionary in the classification model. Finally, the wavelet packet energy features are imported into the sparse dictionary classification model, and the fault diagnosis is completed by outputting the fault categories using the self-driven characteristics of the data. The results show that the fault identification accuracy using the proposed method is 97%. In addition, the experimental validation under different states and working conditions with different rotational speeds allows the superiority and effectiveness of the algorithm in this paper to be tested and has the feasibility of a practical application in engineering.

Model reduction to spectral submanifolds in piecewise smooth dynamical systems

We develop a model reduction technique for piecewise smooth dynamical systems using spectral submanifolds. Specifically, we construct low-dimensional, sparse, nonlinear and piecewise smooth models on unions of slow and attracting spectral submanifolds (SSMs) for each smooth subregion of the phase space and then properly match them. We applied this methodology to both equation-driven and data-driven problems, with and without external forcing.