L1 Regularization Term Research Articles

Target-oriented image-domain elastic least-squares reverse time migration

Elastic least-squares reverse time migration (ELSRTM), as an imaging method, offers advantages over conventional elastic reverse time migration (ERTM), including higher resolution, better amplitude balancing, reduced crosstalk, and broader bandwidth. However, conventional ELSRTM involves iterative processes in the data domain, resulting in high computational costs. Moreover, since time is continuous during data domain extrapolation, it cannot solely focus on the target area within the subsurface medium. In contrast, image-domain ELSRTM (IDELSRTM) exhibits high computational efficiency and the ability to image target area. Currently, research on image-domain least-squares reverse time migration is predominantly focused on the acoustic wave assumption, despite elastic waves being closer to the actual subsurface medium and providing richer imaging information. In this study, within the framework of data domain ELSRTM, we derived the objective function for the IDELSRTM and introduced an L1 regularization term under the L2 norm to enhance inversion stability. We devised an inversion strategy employing the fast iterative shrinkage-thresholding algorithm (FISTA). Furthermore, drawing from the point spread functions theory in optics, we derived the mapping relationship between the elastic multi-parameter point spread functions (PSF) and the elastic multi-parameter Hessian matrix, and the relationship between the Hessian matrix and the ERTM images. We provided the computational method for the elastic multi-parameter Hessian matrix and utilized it as the linearized forward operator for IDELSRTM. Through numerical experiments, we further elucidated the relationship between the ERTM images and the Hessian matrix under the framework of IDELSRTM, along with the sources of crosstalk in ERTM. Applying our proposed target-oriented IDELSRTM method to layered models and the Marmousi2 model, we demonstrated its effectiveness in improving imaging resolution and quality with only a marginal increase in computational overhead compared to conventional ERTM.

Safe dynamic sparse training of modified RBF networks for joint feature selection and classification

Developing an efficient embedded feature selection method for both binary and multiclass classification problems is a fundamental topic to be further studied. In this paper, the conventional radial basis function (RBF) network is modified for joint feature selection and classification. By utilizing the anisotropic Gaussian basis function, an individual weight parameter is assigned to each feature for feature weighting. The L1-regularized loss function is designed by combining the softmax cross entropy loss with L1 regularization terms imposing on feature and output weights. A specialized active-set limited-memory projected Quasi-Newton (SAL-PQN) algorithm is proposed to minimize the L1-regularized loss function, which can optimize all the adjustable parameters and zero out the redundant feature and output weights simultaneously. Furthermore, a safe dynamic pruning strategy is integrated into the SAL-PQN algorithm for dynamic sparse training of the modified RBF network, which continuously excludes the dynamically zeroed-out feature weights from the training process. It is theoretically assured of yielding the same gradient updates as SAL-PQN, thus the relevant computation of the zeroed-out feature weights in the forward and backward propagations can be safely eliminated. The experimental results demonstrate the effectiveness and efficiency of the proposed methods for joint feature selection and classification, and illustrate the practical utility of the inherent feature weighting capability.

Portfolio Optimization with Multi-Trend Objective and Accelerated Quasi-Newton Method

We propose a portfolio optimization method with a multi-trend objective and an accelerated quasi-Newton method (MTO-AQNM). It leverages a BFGS-based quasi-Newton algorithm and incorporates an ℓ1 regularization term and the self-funding constraint. The MTO is designed to identify multiple trend reversals. Different trend reversals are asymmetric, and we hoped to extract rich and effective information from them. The AQNM adopts the BFGS method with the Wolfe conditions, which reduces computational complexity and improves convergence speed. We wanted to evaluate the performance of our algorithm through financial markets that were asymmetric in all respects. To this end, we conducted comprehensive experimental approaches on six benchmark data sets of real-world financial markets that were asymmetric in time, frequency, and asset type. Our method demonstrated superior performance over other state-of-the-art competitors across several mainstream evaluation metrics.

Multi-cluster nonlinear unsupervised feature selection via joint manifold learning and generalized Lasso

Due to the scarcity of data labels, unsupervised feature selection has received a lot of attention in recent years. While many unsupervised feature selection methods are capable of selecting relevant features, they often fail to comprehensively consider the impact of both local and global information of the data on feature selection, nor can they effectively handle the complex nonlinear relationships commonly found in real-world data. As a result, suboptimal feature subsets are often selected. In this paper, inspired by the Uniform Manifold Approximation and Projection (UMAP) manifold learning technique and the nonlinear sparse learning method based on Feature-Wise Kernelized Lasso, we propose a novel unsupervised feature selection method called Multi-Cluster Unsupervised Nonlinear Feature Selection based on UMAP and block HSIC Lasso (MUNFS). MUNFS greatly improves the representation of high-dimensional data during dimensionality reduction and effectively handles complex nonlinear relationships in such data. Specifically, by capturing the intrinsic topology of the data, MUNFS accurately preserves the local structure of the data while keeping as much of the global structure as possible. Furthermore, the kernel-based Hilbert–Schmidt Independence Criterion (HSIC) may measure the nonlinear dependency between the features and the target variables, while applying the l1 regularization term in feature selection to achieve sparsity. This allows for a more precise assessment of the significance of each feature. Extensive experimental results on five benchmark datasets and eight hyperspectral datasets demonstrate that the MUNFS method performs much better than several other feature selection methods.

Efficient construction and convergence analysis of sparse convolutional neural networks

In this paper, a new variant of convolutional neural network (CNN) based on L1 regularization is proposed. The main consideration is to generate a sparse weight matrix, i.e., to generate a sparse neural network model. To some extent, L1 regularization prevents the occurrence of overfitting and effectively improves the learning efficiency of the network. Unfortunately, the L1 regularization term is non-smooth, which leads to the occurrence of oscillations in experiments and poses a great challenge to the theoretical analysis. So, we overcome this drawback by using polishing techniques and rigorously prove the monotonicity of the error function and the strong and weak convergence theorems of the algorithm. Finally, numerical simulations on several data sets support our theoretical results and the superiority of the proposed algorithm.

Augmented flexible Krylov subspace methods with applications to Bayesian inverse problems

This paper presents two new augmented flexible (AF)-Krylov subspace methods, AF-GMRES and AF-LSQR, to compute solutions of large-scale linear discrete ill-posed problems that can be modeled as the sum of two independent random variables, exhibiting smooth and sparse stochastic characteristics respectively. Following a Bayesian modeling approach, this corresponds to adding a covariance-weighted quadratic term and a sparsity enforcing ℓ1 term in the original least-squares minimization scheme. To handle the ℓ1 regularization term, the proposed approach constructs a sequence approximating quadratic problems that are partially solved using augmented flexible Krylov–Tikhonov methods.Compared to other traditional methods used to solve this minimization problem, such as those based on iteratively reweighted norm schemes, the new algorithms build a single (augmented, flexible) approximation (Krylov) subspace that encodes information about the different regularization terms through adaptable “preconditioning”. The solution space is then expanded as soon as a new problem within the sequence is defined. This also allows for the regularization parameters to be chosen on-the-fly at each iteration. Compared to most recent work on generalized flexible Krylov methods [6], our methods offer theoretical assurance of convergence and a more stable numerical performance. The efficiency of the new methods is shown through a variety of experiments, including a synthetic image deblurring problem, a synthetic atmospheric transport problem, and fluorescence molecular tomography reconstructions using both synthetic and real-world experimental data.

Entropy Error-based Convolutional Neural Network Sparse Optimization and Application

Convolutional Neural Networks (CNNs), as quintessential representatives of deep neural networks, have found widespread applications in numerous domains such as image recognition, object detection, and image generation, owing to their robust nonlinear mapping capabilities. However, the practical deployment of CNNs faces challenges such as low learning efficiency and subpar accuracy due to the intricacies in the network structure. This paper introduces the smoothing L1 regularization term atop the conventional entropy loss function, effectively enhancing both the learning efficiency and algorithmic precision of Convolutional Neural Networks. The efficacy of the improved algorithm is substantiated through numerical simulations.

Solving multiscale elliptic problems by sparse radial basis function neural networks

Machine learning has been successfully applied to various fields of scientific computing in recent years. For problems with multiscale features such as flows in porous media and mechanical properties of composite materials, however, it remains difficult for machine learning methods to resolve such multiscale features, especially when the small scale information is present. In this work, we propose a sparse radial basis function neural network method to solve elliptic partial differential equations (PDEs) with multiscale coefficients. Inspired by the deep mixed residual method, we rewrite the second-order problem into a first-order system and employ multiple radial basis function neural networks (RBFNNs) to approximate unknown functions in the system. To avoid the overfitting and improve the performance of RBFNN, an additional regularization is introduced in the loss function. Thus the loss function contains two parts: the L2 loss for the residual of the first-order system and boundary conditions, and the ℓ1 regularization term for the weights of radial basis functions (RBFs). An algorithm for optimizing the specific loss function is introduced to accelerate the training process. The accuracy and effectiveness of the proposed method are demonstrated through a collection of multiscale problems with scale separation, discontinuity and multiple scales from one to three dimensions. Notably, the ℓ1 regularization can achieve the goal of representing the solution by fewer RBFs. As a consequence, the total number of RBFs scales like O(ε−nτ), where ε is the smallest scale, n is the dimensionality, and τ is typically smaller than 1. It is worth mentioning that the proposed method not only has the numerical convergence and thus provides a reliable numerical solution in three dimensions when a classical method is typically not affordable, but also outperforms most other available machine learning methods in terms of accuracy and robustness. Executable files are available at https://github.com/wangzhiwensuda/SRBFNN-for-multiscale-elliptic-problems. The program was implemented in PyTorch and was run on Linux.

On the Adaptive Penalty Parameter Selection in ADMM

Many data analysis problems can be modeled as a constrained optimization problem characterized by nonsmooth functionals, often because of the presence of ℓ1-regularization terms. One of the most effective ways to solve such problems is through the Alternate Direction Method of Multipliers (ADMM), which has been proved to have good theoretical convergence properties even if the arising subproblems are solved inexactly. Nevertheless, experience shows that the choice of the parameter τ penalizing the constraint violation in the Augmented Lagrangian underlying ADMM affects the method’s performance. To this end, strategies for the adaptive selection of such parameter have been analyzed in the literature and are still of great interest. In this paper, starting from an adaptive spectral strategy recently proposed in the literature, we investigate the use of different strategies based on Barzilai–Borwein-like stepsize rules. We test the effectiveness of the proposed strategies in the solution of real-life consensus logistic regression and portfolio optimization problems.

Vibration-based incipient surge detection and diagnosis of the centrifugal compressor using adaptive feature fusion and sparse ensemble learning approach

As a critical high-speed rotating machinery, the centrifugal compressor has been widely used in various modern industries. However, it is subject to a potential damaging phenomenon called surge that can be caused by several factors such as unmatched design, improper operation, inlet and outlet blockage, and so on, which thus may result in catastrophic accidents. This paper concerns the incipient surge detection and diagnosis (ISDD) of the centrifugal compressor based on its bearing vibration signals, leveraging adaptive feature fusion and sparse ensemble learning approach to develop a data-driven-based intelligent diagnostic model. Firstly, vibration signals are decomposed by the empirical mode decomposition (EMD) method, and various features can be extracted from the obtained intrinsic mode functions (IMFs) with rich information of the time series. Next, maximum likelihood estimation (MLE) is utilized to compute the intrinsic dimension of the extracted feature vectors, which are then reduced to the corresponding dimensionality adaptively by kernel principal component analysis (kernel PCA). Coping with the multi-dimensional input and nonlinearly separable classification problem, an ensemble learning approach with L1-regularization term is adopted for incipient surge identification of the centrifugal compressor. A novel and hyperparameter-relaxed hybrid algorithm is employed for the sparse ensemble learning procedure, which guarantees transcendental accuracy and preeminent generalization capability while ensuring high robustness and sparsity of the intelligent model. Finally, in addition to diagnostic tasks of the centrifugal compressor datasets under different working conditions, the ISDD performance of the proposed method in imbalanced sample scenarios, varying working conditions and additional noise interference circumstances is comprehensively investigated, and the overall success indicates the robustness and superiority of the proposed method compared with other existing approaches.

Improved TV Image Denoising over Inverse Gradient

Noise in an image can affect one’s extraction of image information, therefore, image denoising is an important image pre-processing process. Many of the existing models have a large number of estimated parameters, which increases the time complexity of the model solution and the achieved denoising effect is less than ideal. As a result, in this paper, an improved image-denoising algorithm is proposed based on the TV model, which effectively solves the above problems. The L1 regularization term can make the solution generated by the model sparser, thus facilitating the recovery of high-quality images. Reducing the number of estimated parameters, while using the inverse gradient to estimate the regularization parameters, enables the parameters to achieve global adaption and improves the denoising effect of the model in combination with the TV regularization term. The split Bregman iteration method is used to decouple the model into several related subproblems, and the solutions of the coordinated subproblems are derived as optimal solutions. It is also shown that the solution of the model converges to a Karush–Kuhn–Tucker point. Experimental results show that the algorithm in this paper is more effective in both preserving image texture structure and suppressing image noise.

Distributed sparse identification for stochastic dynamic systems under cooperative non-persistent excitation condition

This paper considers the distributed sparse identification problem over wireless sensor networks such that all sensors cooperatively estimate the unknown sparse parameter vector of stochastic dynamic systems by using the local information from neighbors. A distributed sparse least squares algorithm is proposed by minimizing a local information criterion formulated as a linear combination of accumulative local estimation error and L1-regularization term. The upper bound of the estimation error of the proposed algorithm is presented. Furthermore, by designing a suitable adaptive weighting coefficient based on the local observation data, the set convergence of zero elements with a finite number of observations is obtained under a cooperative non-persistent excitation condition. It is shown that the proposed distributed algorithm can work well in a cooperative way even though none of the individual sensors can fulfill the estimation task. Our theoretical results are obtained without relying on the independency assumptions of regression signals that have been commonly used in the existing literature. Thus, our results are expected to be applied to stochastic feedback systems. Finally, the numerical simulations are provided to demonstrate the effectiveness of our theoretical results.

Flexible quasi-2D inversion of time-domain AEM data, using a wavelet-based complexity measure

SUMMARYRegularization methods improve the stability of ill-posed inverse problems by introducing some a priori characteristics for the solution such as smoothness or sharpness. In this contribution, we propose a multidimensional scale-dependent wavelet-based ℓ1-regularization term to cure the ill-posedness of the airborne (time-domain) electromagnetic induction inverse problem. The regularization term is flexible, as it can recover blocky, smooth and tunable in-between inversion models, based on a suitable wavelet basis function. For each orientation, a different wavelet basis function can be used, introducing an additional relative regularization parameter. We propose a calibration method to determine (an educated initial guess for) this relative regularization parameter, which reduces the need to optimize for this parameter and, consequently, the overall computation time is under control. We apply our novel scheme to a time-domain airborne electromagnetic data set in Belgian saltwater intrusion context, but the scheme could equally apply to any other 2D or 3D geophysical inverse problem.

Batch Gradient Learning Algorithm with Smoothing L1 Regularization for Feedforward Neural Networks

Regularization techniques are critical in the development of machine learning models. Complex models, such as neural networks, are particularly prone to overfitting and to performing poorly on the training data. L1 regularization is the most extreme way to enforce sparsity, but, regrettably, it does not result in an NP-hard problem due to the non-differentiability of the 1-norm. However, the L1 regularization term achieved convergence speed and efficiency optimization solution through a proximal method. In this paper, we propose a batch gradient learning algorithm with smoothing L1 regularization (BGSL1) for learning and pruning a feedforward neural network with hidden nodes. To achieve our study purpose, we propose a smoothing (differentiable) function in order to address the non-differentiability of L1 regularization at the origin, make the convergence speed faster, improve the network structure ability, and build stronger mapping. Under this condition, the strong and weak convergence theorems are provided. We used N-dimensional parity problems and function approximation problems in our experiments. Preliminary findings indicate that the BGSL1 has convergence faster and good generalization abilities when compared with BGL1/2, BGL1, BGL2, and BGSL1/2. As a result, we demonstrate that the error function decreases monotonically and that the norm of the gradient of the error function approaches zero, thereby validating the theoretical finding and the supremacy of the suggested technique.

Automatic CT whole-lung segmentation in radiomics discrimination: Methodology and application in pneumonia diagnosis and distinguishment

Radiomics based on lesion segmentation has been widely accepted for disease diagnosis; however, it is difficult to precisely determine the boundary for pneumonia due to its diffuse characteristics. In this study, we aimed to propose an automatic radiomics method using whole-lung segmentation in pneumonia discrimination and assist clinical practitioners in fast and accurate diagnosis. In the discovery set, data from 151 participants diagnosed with type A or B influenza virus pneumonia, 63 diagnosed with coronavirus disease 2019 (COVID-19) and 50 healthy participants were collected. The three groups of data were compared in pairs. A total of 117 radiomics features were extracted from whole-lung images segmented by a four-layer U-net. We then utilized a logistic regression model to train the model and used the area under the receiver operating characteristic curve (AUC) to assess its performance. The L1 regularization term was used in feature selection, and 10-fold cross-validation was used to tune the hyperparameters. Fourteen radiomics features were selected to classify influenza pneumonia and health, and the AUC was 0.957 (95% confidential interval (CI): 0.939, 0.976) in the training set and 0.914 (95% CI: 0.866, 0.963) in the testing set. Eighteen features were selected for COVID-19 and health, and the AUC was 0.949 (95% CI: 0.926, 0.973) in the training set and 0.911 (95% CI: 0.859, 0.963) in the testing set. Twenty-eight features were selected for influenza virus pneumonia and COVID-19, and the AUC was 0.895 (95% CI: 0.870, 0.920) in the training set and 0.839 (95% CI: 0.791, 0.887) in the testing set. The results show that the automatic radiomics model based on whole lung segmentation is effective in distinguishing influenza virus pneumonia, COVID-19 and health, and may assist in the diagnosis of influenza virus pneumonia and COVID-19.

RgCop-A regularized copula based method for gene selection in single-cell RNA-seq data.

Gene selection in unannotated large single cell RNA sequencing (scRNA-seq) data is important and crucial step in the preliminary step of downstream analysis. The existing approaches are primarily based on high variation (highly variable genes) or significant high expression (highly expressed genes) failed to provide stable and predictive feature set due to technical noise present in the data. Here, we propose RgCop, a novel regularized copula based method for gene selection from large single cell RNA-seq data. RgCop utilizes copula correlation (Ccor), a robust equitable dependence measure that captures multivariate dependency among a set of genes in single cell expression data. We formulate an objective function by adding l1 regularization term with Ccor to penalizes the redundant co-efficient of features/genes, resulting non-redundant effective features/genes set. Results show a significant improvement in the clustering/classification performance of real life scRNA-seq data over the other state-of-the-art. RgCop performs extremely well in capturing dependence among the features of noisy data due to the scale invariant property of copula, thereby improving the stability of the method. Moreover, the differentially expressed (DE) genes identified from the clusters of scRNA-seq data are found to provide an accurate annotation of cells. Finally, the features/genes obtained from RgCop is able to annotate the unknown cells with high accuracy.

Machine Learning Prediction of Liver Allograft Utilization From Deceased Organ Donors Using the National Donor Management Goals Registry.

Several machine learning classifiers were trained to predict transplantation of a liver graft. We utilized 127 variables available in the DMG dataset. We included data from potential deceased organ donors between April 2012 and January 2019. The outcome was defined as liver recovery for transplantation in the operating room. The prediction was made based on data available 12-18 h after the time of authorization for transplantation. The data were randomly separated into training (60%), validation (20%), and test sets (20%). We compared the performance of our models to the Liver Discard Risk Index. Of 13 629 donors in the dataset, 9255 (68%) livers were recovered and transplanted, 1519 recovered but used for research or discarded, 2855 were not recovered. The optimized gradient boosting machine classifier achieved an area under the curve of the receiver operator characteristic of 0.84 on the test set, outperforming all other classifiers. This model predicts successful liver recovery for transplantation in the operating room, using data available early during donor management. It performs favorably when compared to existing models. It may provide real-time decision support during organ donor management and transplant logistics.

Sdgan: Improve Speech Enhancement Quality by Information Filter

The speech denoising model based on adversarial generative network has achieved better results than the traditional machine learning model. In this paper, for the short cut connection in the generator, we discuss its influence on the information transfer between encoder and decoder, and propose SDGAN at target. SDGAN sets linear and convolution filters in the short cut connection which adaptively learn the optimal information processing. The information filter still enables the generator to solve the gradient vanishing problem, and it can also avoid information redundancy and improve expression ability. In addition, SDGAN replaces the L1 regularization term in loss function with the L2 regularization term, which not only makes the output speech of the generator closer to the clean speech, but also avoids sparsity. In the experiments, SDGAN significantly performs better than other traditional GAN in five performance metrics (such as PESQ), and the effect of convolution filter is better than that of linear filter.

Generating Cartoon Images from Face Photos with Cycle-Consistent Adversarial Networks

The generative adversarial network (GAN) is first proposed in 2014, and this kind of network model is machine learning systems that can learn to measure a given distribution of data, one of the most important applications is style transfer. Style transfer is a class of vision and graphics problems where the goal is to learn the mapping between an input image and an output image. CYCLE-GAN is a classic GAN model, which has a wide range of scenarios in style transfer. Considering its unsupervised learning characteristics, the mapping is easy to be learned between an input image and an output image. However, it is difficult for CYCLE-GAN to converge and generate high-quality images. In order to solve this problem, spectral normalization is introduced into each convolutional kernel of the discriminator. Every convolutional kernel reaches Lipschitz stability constraint with adding spectral normalization and the value of the convolutional kernel is limited to [0, 1], which promotes the training process of the proposed model. Besides, we use pretrained model (VGG16) to control the loss of image content in the position of l1 regularization. To avoid overfitting, l1 regularization term and l2 regularization term are both used in the object loss function. In terms of Frechet Inception Distance (FID) score evaluation, our proposed model achieves outstanding performance and preserves more discriminative features. Experimental results show that the proposed model converges faster and achieves better FID scores than the state of the art.

Detecting Meaningful Clusters From High-Dimensional Data: A Strongly Consistent Sparse Center-Based Clustering Approach.

In context to high-dimensional clustering, the concept of feature weighting has gained considerable importance over the years to capture the relative degrees of importance of different features in revealing the cluster structure of the dataset. However, the popular techniques in this area either fail to perform feature selection or do not preserve the simplicity of Lloyd's heuristic to solve the k-means problem and the like. In this paper, we propose a Lasso Weighted k-means ( LW- k-means) algorithm, as a simple yet efficient sparse clustering procedure for high-dimensional data where the number of features ( p) can be much higher than the number of observations ( n). The LW- k-means method imposes an l1 regularization term involving the feature weights directly to induce feature selection in a sparse clustering framework. We develop a simple block-coordinate descent type algorithm with time-complexity resembling that of Lloyd's method, to optimize the proposed objective. In addition, we establish the strong consistency of the LW- k-means procedure. Such an analysis of the large sample properties is not available for the conventional sparse k-means algorithms, in general. LW- k-means is tested on a number of synthetic and real-life datasets and through a detailed experimental analysis, we find that the performance of the method is highly competitive against the baselines as well as the state-of-the-art procedures for center-based high-dimensional clustering, not only in terms of clustering accuracy but also with respect to computational time.