Regularization Term Research Articles

Self-supervised tomographic image noise suppression via residual image prior network

Computed tomography (CT) denoising is a challenging task in medical imaging that has garnered considerable attention. Supervised networks require a lot of noisy-clean image pairs, which are always unavailable in clinical settings. Existing self-supervised algorithms for suppressing noise with paired noisy images have limitations, such as ignoring the residual between similar image pairs during training and insufficiently learning the spectrum information of images. In this study, we propose a Residual Image Prior Network (RIP-Net) to sufficiently model the residual between the paired similar noisy images. Our approach offers new insights into the field by addressing the limitations of existing methods. We first establish a mathematical theorem clarifying the non-equivalence between similar-image-based self-supervised learning and supervised learning. It helps us better understand the strengths and limitations of self-supervised learning. Secondly, we introduce a novel regularization term to model a low-frequency residual image prior. This can improve the accuracy and robustness of our model. Finally, we design a well-structured denoising network capable of exploring spectrum information while simultaneously sensing context messages. The network has dual paths for modeling high and low-frequency compositions in the raw noisy image. Additionally, context perception modules capture local and global interactions to produce high-quality images. The comprehensive experiments on preclinical photon-counting CT, clinical brain CT, and low-dose CT datasets, demonstrate that our RIP-Net is superior to other unsupervised denoising methods.

Extract Implicit Semantic Friends and Their Influences from Bipartite Network for Social Recommendation

Social recommendation often incorporates trusted social links with user-item interactions to enhance rating prediction. Although methods that aggregate explicit social links have shown promising prospects, they are often constrained by the absence of explicit social data and the assumption of homogeneity, thus overlooking variations in social influence and consistency. These limitations hinder semantic expression and recommendation performance. Therefore, we propose a novel framework for social recommendation. First, we design a bipartite network embedding scheme, which learns vertex representations in the embedding space by modeling 1st-order explicit relations and higher-order implicit relations between vertices. Then, the similarity of the embedding vectors is used to extract top-k semantically consistent friends for each user. Next, we design an algorithm to assign a specific influence value to each user. Finally, we combine the top-k friends of the user and their influence values into an ensemble and add it as a regularization term to the rating prediction process of the user to correct the bias. Experiments on three real benchmark datasets show significant improvements in EISF over state-of-the-art methods.

PatchSkip: A lightweight technique for effectively alleviating over-smoothing in vision transformers

Recently vision transformers (ViTs) have encountered the over-smoothing problem, which reduces their capacity by mapping input patches into a similar latent representation. Existing methods introduce regularization terms to alleviate over-smoothing but often increase computational costs. To address this, this paper proposes PatchSkip, a novel and flexible dropout variant, alleviating the over-smoothing problem of ViTs in a lightweight manner. Specifically, PatchSkip draws inspiration from the fact that a similar over-smoothing problem in GNNs is primarily caused by static adjacent matrices leading to solitary message passing mode between nodes. PatchSkip constructs graphs with patch embeddings and analyzes the adjacent matrices in ViTs. By randomly selecting specific patch embeddings to bypass transformer blocks, PatchSkip is proved to generate various adjacent matrices and acts as a multi-mode message passing engine, providing diverse modes of message passing between patches. The effectiveness of PatchSkip in preventing over-smoothing is demonstrated through theoretical proofs and empirical visualizations. Furthermore, PatchSkip is evaluated on various datasets and backbones, showing significant performance improvements while reducing computational costs. For example, when trained on Tiny-ImageNet from scratch, PatchSkip improves the performance of the vanilla CrossViT by 3.85% while reducing computational costs by over 20%.

Graph-based semi-supervised learning with non-convex graph total variation regularization

Graph total variation (GTV) is a widely employed regularization in graph-based semi-supervised learning (GSSL), which enforce the piece-wise smoothness of the label values concerning the underlying graph structure. However, due to its inherently biased estimation, GTV regularization tends to cause an underestimation of label values on boundary nodes . In this paper, we propose a novel GSSL model with non-convex GTV regularization. Specifically, we construct the non-convex GTV regularization by subtracting the generalized Moreau envelope from the original GTV regularization term, which can reduce the estimation bias and thus effectively estimate the label values of the boundary nodes. We demonstrate that under certain conditions of the non-convex control parameter, our proposed GSSL model maintains global convexity, thereby providing a theoretical guarantee for algorithm convergence. Additionally, we present an efficient alternating direction multiplier method (ADMM) to solve the proposed model. Finally, we validate the effectiveness on both synthetic data and background subtraction. The proposed GSSL model outperforms state-of-the-art model with a 17.97% improvement in SNR value on synthetic data, and increases average recall, average precision, and average F-measure by 4.92%, 1.19%, and 1.57% respectively on the pan–tilt-zoom(PTZ) challenge of background subtraction.

Safe dynamic sparse training of modified RBF networks for joint feature selection and classification

Developing an efficient embedded feature selection method for both binary and multiclass classification problems is a fundamental topic to be further studied. In this paper, the conventional radial basis function (RBF) network is modified for joint feature selection and classification. By utilizing the anisotropic Gaussian basis function, an individual weight parameter is assigned to each feature for feature weighting. The L1-regularized loss function is designed by combining the softmax cross entropy loss with L1 regularization terms imposing on feature and output weights. A specialized active-set limited-memory projected Quasi-Newton (SAL-PQN) algorithm is proposed to minimize the L1-regularized loss function, which can optimize all the adjustable parameters and zero out the redundant feature and output weights simultaneously. Furthermore, a safe dynamic pruning strategy is integrated into the SAL-PQN algorithm for dynamic sparse training of the modified RBF network, which continuously excludes the dynamically zeroed-out feature weights from the training process. It is theoretically assured of yielding the same gradient updates as SAL-PQN, thus the relevant computation of the zeroed-out feature weights in the forward and backward propagations can be safely eliminated. The experimental results demonstrate the effectiveness and efficiency of the proposed methods for joint feature selection and classification, and illustrate the practical utility of the inherent feature weighting capability.

Unveiling SAR target recognition networks: Adaptive Perturbation Interpretation for enhanced understanding

Deep neural networks (DNNs) have obtained remarkable achievements in various vision tasks. However, DNNs’ mechanism remains obscure, especially in synthetic aperture radar (SAR) target recognition. To clearly understand DNNs’ mechanism, some interpretation algorithms are proposed to provide a saliency map to visualize DNN’s decisions, which can be categorized into (1) propagation-based methods, (2) activation-based methods, and (3) perturbation-based methods. Unfortunately, each of them has some imperfection in producing saliency maps. The saliency maps produced by propagation-based methods are sensitive to SAR speckles. Activation-based methods have difficulty producing a saliency map where the highlighted region covers the entire feature. Perturbation-based methods tend to forfeit intricate details of the target in saliency map during optimization. To mitigate the above limitations, this paper proposed Adaptive Perturbation Interpretation (API), with two aims: (1) to improve the interpretability of SAR recognition networks and (2) to recover the fine-grained lost during the optimization procedures of perturbation-based methods. Specifically, we first introduce a Multi-information Perturbation Optimization Module (MPOM) to retain the target’s local characteristics while constraining saliency map’s sparsity by configuring regularization terms. Additionally, we further introduce a Feature Compensation Module (FCM) to recover the fine-grained feature lost during the optimization process in MPOM. Extensive experiments are implemented on SAR interpretation benchmark datasets, MSTAR and SARAIRcraft1.0, the results demonstrate the superiority of API to other state-of-the-art methods.

Fuel-Optimal Trajectory Planning for Lunar Vertical Landing

In this paper we consider a trajectory planning problem arising from a lunar vertical landing with minimum fuel consumption. The vertical landing requirement is written as a final steering angle constraint, and a nonnegative regularization term is proposed to modify the cost functional. In this way, the final steering angle constraint will be inherently satisfied according to Pontryagin’s Minimum Principle. As a result, the modified optimal steering angle has to be determined by solving a transcendental equation. To this end, a transforming procedure is employed, which allows for finding the desired optimal steering angle by a simple bisection method. Consequently, the vertical landing trajectory can be found offline by the indirect shooting method. Finally, some numerical examples are presented to demonstrate and verify the developments of the paper.

A Nesterov Type Algorithm with Double Tikhonov Regularization: Fast Convergence of the Function Values and Strong Convergence to the Minimal Norm Solution

We investigate the strong convergence properties of a Nesterov type algorithm with two Tikhonov regularization terms in connection to the minimization problem of a smooth convex function f. We show that the generated sequences converge strongly to the minimal norm element from argminf\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\ ext {argmin}f$$\\end{document}. We also show fast convergence for the potential energies f(xn)-minf\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$f(x_n)-\ ext {min}f$$\\end{document} and f(yn)-minf\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$f(y_n)-\ ext {min}f$$\\end{document}, where (xn),(yn)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$(x_n),\\,(y_n)$$\\end{document} are the sequences generated by our algorithm. Further we obtain fast convergence to zero of the discrete velocity and some estimates concerning the value of the gradient of the objective function in the generated sequences. Via some numerical experiments we show that we need both Tikhonov regularization terms in our algorithm in order to obtain the strong convergence of the generated sequences to the minimum norm minimizer of our objective function.

On the complexity of a quadratic regularization algorithm for minimizing nonsmooth and nonconvex functions

In this paper, we consider the problem of minimizing the function f ( x ) = g 1 ( x ) + g 2 ( x ) − h ( x ) over R n , where g 1 is a proper and lower semicontinuous function, g 2 is continuously differentiable with a Hölder continuous gradient and h is a convex function that may be nondifferentiable. This problem has important practical applications but is challenging to solve due to the presence of nonconvexities and nonsmoothness. To address this issue, we propose an algorithm based on a proximal gradient method that uses a quadratic approximation of the function g 2 and a nonconvex regularization term. We show that the number of iterations required to reach our stopping criterion is O ( max { ϵ − β + 1 β , η 2 β ϵ − 2 ( β + 1 ) β } ) . Our approach offers a promising strategy for solving this challenging optimization problem and has potential applications in various fields. Numerical examples are provided to illustrate the theoretical results.

DMTFS-FO: Dynamic multi-task feature selection based on flexible loss and orthogonal constraint

Multi-task feature selection (MTFS) has been proven effective for reducing the curse of dimensionality in large-scale classification. Many existing MTFS methods assume that all tasks learn concurrently in a static environment without considering the dynamism of tasks in real-world scenarios. However, new tasks emerge dynamically in practical applications, meaning that the aforementioned static assumption is insufficient. In this paper, we construct a dynamic multi-task feature selection framework to achieve feature reduction for constantly arriving new tasks. First, we modify the traditional mapping by shifting from hard labels to soft labels. Unlike the conventional rigid mapping, the new flexible loss function changes the direct mapping strategy to an indirect one. Second, we use the orthogonal regularization term to constrain the independent relationship between new and old tasks. This ensures that the selected relevant features for new tasks differ from prior tasks. Finally, we integrate the flexible loss and the orthogonal regularization term in the dynamic multi-task feature selection framework. Our method outperforms nine other advanced feature selection methods in terms of effectiveness and efficiency across six datasets. For example, the ACC value of our method is almost 1% higher than the next-best method on the large-scale SUN dataset.

Total Third-Degree Variation for Noise Reduction in Atomic-Resolution STEM Images.

Scanning Transmission Electron Microscopy (STEM) enables direct determination of atomic arrangements in materials and devices. However, materials such as battery components are weak for electron beam irradiation and low electron doses are required to prevent beam-induced damages. Noise removal is thus essential for precise structural analysis of electron beam sensitive materials at atomic resolution. Total square variation (TSV) regularization is an algorithm that exhibits high noise removal performance. However, the use of the TSV regularization term leads to significant image blurring and intensity reduction. To address these problems, we here propose a new approach adopting L2 norm regularization based on higher-order total variation. An atomic-resolution STEM image can be approximated as a set of smooth curves represented by quadratic functions. Since the third-degree derivative of any quadratic function is 0, total third-degree variation (TTDV) is suitable for a regularization term. The application of TTDV for denoising the atomic-resolution STEM image of CaF2 observed along the [001] zone axis is shown, where we can clearly see the Ca and F atomic columns without compromising image quality.

Generalized div-curl based regularization for physically constrained deformable image registration

Variational image registration methods commonly employ a similarity metric and a regularization term that renders the minimization problem well-posed. However, many frequently used regularizations such as smoothness or curvature do not necessarily reflect the underlying physics that apply to anatomical deformations. This, in turn, can make the accurate estimation of complex deformations particularly challenging. Here, we present a new highly flexible regularization inspired from the physics of fluid dynamics which allows applying independent penalties on the divergence and curl of the deformations and/or their nth order derivative. The complexity of the proposed generalized div-curl regularization renders the problem particularly challenging using conventional optimization techniques. To this end, we develop a transformation model and an optimization scheme that uses the divergence and curl components of the deformation as control parameters for the registration. We demonstrate that the original unconstrained minimization problem reduces to a constrained problem for which we propose the use of the augmented Lagrangian method. Doing this, the equations of motion greatly simplify and become managable. Our experiments indicate that the proposed framework can be applied on a variety of different registration problems and produce highly accurate deformations with the desired physical properties.

Distributed ISAR imaging based on convolution and total variation reweighted l1 regularization

ABSTRACT Distributed Inverse Synthetic Aperture Radar (ISAR) systems employ multiple spatially separated radars to observe a target, with the potential to enhance radar imaging azimuth resolution and gather more target information. Due to the existence of gaps in the observation angles of each radars, the coherence between pulses of radar echo is disrupted during fusion imaging, making it challenging for the Range-Doppler (RD) algorithm to achieve well-focused images. This paper proposes an imaging method based on sparse signal recovery (SSR), using the reweighted l 1 norm with convolution and total variation as regularization terms. This approach not only yields well-focused ISAR images but also better preserves the overall target information compared to other methods, while also exhibiting excellent noise resistance. In addition, the alternating direction method of multipliers (ADMM) algorithm is utilized for the solution to reduce computational complexity. Simulation and measured data verify the effectiveness of the proposed method.

Poisson tensor completion with transformed correlated total variation regularization

Tensor completion involves recovering the underlying tensor from partial observations, and in this paper we focus on the point that these observations obey the Poisson distribution. To contend with this problem, we adopt a popular method that minimizes the sum of the data-fitting term and the regularization term under a uniform sampling mechanism. Specifically, we consider the negative logarithmic maximum likelihood estimate of the Poisson distribution as the data-fitting term. To effectively characterize the intrinsic structure of the tensor data, we propose a parameter-free regularization term that can simultaneously capture the low rankness and local smoothness of the underlying tensor. Here, the transformed tensor nuclear norm is used to explore the low rankness under suitable unitary transformations. We present theoretical derivations to demonstrate the feasibility of the proposed model. Furthermore, we develop an algorithm based on the alternating direction multiplier method (ADMM) to efficiently solve the proposed optimization problem, with its overall convergence being established. A series of numerical experiments show that proposed model yields a pleasing accuracy over several state-of-the-art models.

Discontinuous optimal operator for the P1 Galerkin method approximation of elliptic PDEs with a weakly regular source term

Discontinuous optimal operator for the P1 Galerkin method approximation of elliptic PDEs with a weakly regular source term

Label-Free Poisoning Attack Against Deep Unsupervised Domain Adaptation

Deep unsupervised domain adaptation (UDA) has significantly boosted the performance of deep models on different domains by transferring knowledge from a source domain to a target domain. However, its robustness against adversarial attacks has not been explored due to the challenges of highly non-convex deep models and different data distribution. In this paper, we give the first attempt to analyze the vulnerability of deep UDA and propose a label-free poisoning attack (LFPA), which injects poisoning data into the training data to mislead adaptation between the two domains without ground truth in target domain. Specifically, we design an unsupervised adversarial loss as the attack goal, in which the pseudo-labels are used to approximate the ground-truth. Since retraining the model will gradually degrade the attack performance, we also add a regularization term to the unsupervised loss, which eliminates negative interactions between the training goal and the attack goal. To accelerate the craft of poisons, we select influential samples as the initial poisons and propose a fast reverse-mode optimization method which updates poisons according to the approximate truncated gradients. Experimental results on multiple state-of-the-art deep UDA methods demonstrate the effectiveness of the proposed LFPA and the high sensitivity of UDA to poisoning attacks.

Hyperspectral Anomaly Detection Based on Adaptive Low-Rank Transformed Tensor.

Hyperspectral anomaly detection, which is aimed at distinguishing anomaly pixels from the surroundings in spatial features and spectral characteristics, has attracted considerable attention due to its various applications. In this article, we propose a novel hyperspectral anomaly detection algorithm based on adaptive low-rank transform, in which the input hyperspectral image (HSI) is divided into a background tensor, an anomaly tensor, and a noise tensor. To take full advantage of the spatial-spectral information, the background tensor is represented as the product of a transformed tensor and a low-rank matrix. The low-rank constraint is imposed on frontal slices of the transformed tensor to depict the spatial-spectral correlation of the HSI background. Besides, we initialize a matrix with predefined size and then minimize its l2.1 -norm to adaptively derive an appropriate low-rank matrix. The anomaly tensor is constrained with the l2.1.1 -norm to depict the group sparsity of anomalous pixels. We integrate all regularization terms and a fidelity term into a non-convex problem and develop a proximal alternating minimization (PAM) algorithm to solve it. Interestingly, the sequence generated by the PAM algorithm is proven to converge to a critical point. Experimental results conducted on four widely used datasets demonstrate the superiority of the proposed anomaly detector over several state-of-the-art methods.

Noise suppression for an aeromagnetic measurement system on an unmanned helicopter.

An unmanned helicopter is one of the main platforms for conducting unmanned aerial vehicle aeromagnetic measurements and combines the advantages of rotary-wing and fixed-wing unmanned aerial vehicles. However, unmanned helicopter-based aeromagnetic measurement systems face complex static magnetic noise and maneuvering magnetic interference, which limit their practical performance. To address this issue, an improved multi-channel frequency measurement algorithm for the optically pumped magnetic sensor is proposed to suppress the static magnetic noise proportional to the frequency noise generated by the random quantization error and the airborne electromagnetic interference. A novel aeromagnetic compensation method for the maneuvering magnetic interference is then proposed to weaken the negative effects of the strong multicollinearity of the attitude parameters of the unmanned helicopter on the compensation accuracy and stability by introducing a regularization term and weight matrix. In addition, dedicated software is developed for the real-time calculation and compensation of magnetic interference fields. A dedicated unmanned-helicopter-based aeromagnetic measurement system is developed, and ground and flight experiments are carried out. The ground test results indicate that the static noise of the proposed system is only 0.000 82 nT. In the flight experiments, the system achieves an improvement ratio of 8.33, which is higher than the improvement ratio of 4.37 for a state-of-the-art commercial compensator. Furthermore, the dynamic noise after compensation decreases by 37.6% from 0.0157 to 0.0098 nT.

SPD Manifold Deep Metric Learning for Image Set Classification.

By characterizing each image set as a nonsingular covariance matrix on the symmetric positive definite (SPD) manifold, the approaches of visual content classification with image sets have made impressive progress. However, the key challenge of unhelpfully large intraclass variability and interclass similarity of representations remains open to date. Although, several recent studies have mitigated the two problems by jointly learning the embedding mapping and the similarity metric on the original SPD manifold, their inherent shallow and linear feature transformation mechanism are not powerful enough to capture useful geometric features, especially in complex scenarios. To this end, this article explores a novel approach, termed SPD manifold deep metric learning (SMDML), for image set classification. Specifically, SMDML first selects a prevailing SPD manifold neural network (SPDNet) as the backbone (encoder) to derive an SPD matrix nonlinear representation. To counteract the degradation of structural information during multistage feature embedding, we construct a Riemannian decoder at the end of the encoder, trained by a reconstruction error term (RT), to induce the generated low-dimensional feature manifold of the hidden layer to capture the pivotal information about the visual data describing the imaged scene. We demonstrate through theory and experiments that it is feasible to replace the Riemannian metric with Euclidean distance in RT. Then, the ReCov layer is introduced into the established Riemannian network to regularize the local statistical information within each input feature matrix, which enhances the effectiveness of the learning process. The theoretical analysis of the activation function used in the ReCov layer in terms of continuity and conditions for generating positive definite matrices is beneficial for network design. Inspired by the fact that the single cross-entropy loss used for training is unable to effectively parse the geometric distribution of the deep representations, we finally endow the suggested model with a novel metric learning regularization term. By explicitly incorporating the encoding and processing of the data variations into the network learning process, this term can not only derive a powerful Riemannian representation but also train an effective classifier. The experimental results show the superiority of the proposed approach on three typical visual classification tasks.

Iterative algorithm using decoupling method for third-order tensor deblurring

The present paper is concerned with exploiting an iterative decoupling algorithm to address the problem of third-order tensor deblurring. The regularized deblurring problem, which is mathematically given by the sum of a fidelity term and a regularization term, is decoupled into an observation fidelity and a denoiser model steps. One basic advantage of the iterative decoupling algorithm is that the deblurring problem is supervised by the efficiency of the denoiser model. Thus, we consider a patch-based weighted low-rank tensor with sparsity prior. Numerical tests to image deblurring are given to demonstrate the efficiency of the proposed decoupling based algorithm.