Regularization-based Methods Research Articles

Sample Inflation Interpolation for Consistency Regularization in Remote Sensing Change Detection

Semi-supervised learning has gained significant attention in the field of remote sensing due to its ability to effectively leverage both a limited number of labeled samples and a large quantity of unlabeled data. An effective semi-supervised learning approach utilizes unlabeled samples to enforce prediction consistency under minor perturbations, thus reducing the model’s sensitivity to noise and suppressing false positives in change-detection tasks. This principle underlies consistency regularization-based methods. However, while these methods enhance noise robustness, they also risk overlooking subtle but meaningful changes, leading to information loss and missed detections. To address this issue, we introduce a simple yet efficient method called Sample Inflation Interpolation (SII). This method leverages labeled sample pairs to mitigate the information loss caused by consistency regularization. Specifically, we propose a novel data augmentation strategy that generates additional change samples by combining existing supervised change samples with calculated proportions of change areas. This approach increases both the quantity and diversity of change samples in the training set, effectively compensating for potential information loss and reducing missed detections. Furthermore, to prevent overfitting, small perturbations are applied to the generated sample pairs and their labels. Experiments conducted on two public change detection (CD) datasets validate the effectiveness of our proposed method. Remarkably, even with only 5% of labeled training data, our method achieves performance levels that closely approach those of fully supervised learning models.

Defying catastrophic forgetting via influence function

Deep-learning models need to continually accumulate knowledge from tasks, given that the number of tasks are increasing overwhelmingly as the digital world evolves. However, standard deep-learning models are prone to forgetting about previously acquired skills when learning new ones. Fortunately, this catastrophic forgetting problem can be solved by means of continual learning. One popular approach in this vein is regularization-based method which penalizes parameters by giving their importance. However, a formal definition of parameter importance and theoretical analysis of regularization-based methods are elements that remain under-explored. In this paper, we first rigorously define the parameter importance by influence function, then unify the seminal methods (i.e., EWC, SI and MAS) into one whole framework. Two key theoretical results are presented in this work, and extensive experiments are conducted on standard benchmarks, which verify the superior performance of our proposed method.

Overcoming Catastrophic Forgetting in Continual Learning by Exploring Eigenvalues of Hessian Matrix.

Neural networks tend to suffer performance deterioration on previous tasks when they are applied to multiple tasks sequentially without access to previous data. The problem is commonly known as catastrophic forgetting, a significant challenge in continual learning (CL). To overcome the catastrophic forgetting, regularization-based CL methods construct a regularization-based term, which can be considered as the approximation loss function of previous tasks, to penalize the update of parameters. However, the rigorous theoretical analysis of regularization-based methods is limited. Therefore, we theoretically analyze the forgetting and the convergence properties of regularization-based methods. The theoretical results demonstrate that the upper bound of the forgetting has a relationship with the maximum eigenvalue of the Hessian matrix. Hence, to decrease the upper bound of the forgetting, we propose eiGenvalues ExplorAtion Regularization-based (GEAR) method, which explores the geometric properties of the approximation loss of prior tasks regarding the maximum eigenvalue. Extensive experimental results demonstrate that our method mitigates catastrophic forgetting and outperforms existing regularization-based methods.

Regularization-based methods for ordinal quantification

Quantification, i.e., the task of predicting the class prevalence values in bags of unlabeled data items, has received increased attention in recent years. However, most quantification research has concentrated on developing algorithms for binary and multi-class problems in which the classes are not ordered. Here, we study the ordinal case, i.e., the case in which a total order is defined on the set of n>2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$n>2$$\\end{document} classes. We give three main contributions to this field. First, we create and make available two datasets for ordinal quantification (OQ) research that overcome the inadequacies of the previously available ones. Second, we experimentally compare the most important OQ algorithms proposed in the literature so far. To this end, we bring together algorithms proposed by authors from very different research fields, such as data mining and astrophysics, who were unaware of each others’ developments. Third, we propose a novel class of regularized OQ algorithms, which outperforms existing algorithms in our experiments. The key to this gain in performance is that our regularization prevents ordinally implausible estimates, assuming that ordinal distributions tend to be smooth in practice. We informally verify this assumption for several real-world applications.

Regularization techniques and inverse approaches in 3D Traction Force Microscopy

The conception of inverse methods in the context of Traction Force Microscopy (TFM) is influenced by multiple factors, such as nonlinear effects, dimensionality (2D/3D), and regularization/constraints, amongst others. Solving the inverse problem often requires the inversion of a matrix, and the presence of noise in the measured displacements can lead to unrealistic reconstructed tractions. To address this issue, Tikhonov regularization is commonly used, penalizing high norm values of computed tractions. The aim of this study is to compare the performance of different inverse methodologies (including some original variations) in 3D TFM, considering constraint imposition and regularization along the formulation. The different methodologies are numerically elaborated within a novel combined Newton–Raphson/Finite Element Method scheme that provides converged solutions in few iterations. The impact of constraint imposition and regularization in traction reconstruction is evaluated in terms of accuracy, efficiency (CPU time), and inherent characteristics of the methodology. Results show that, applying regularization and constraints (based on the fulfillment of fundamental principles) provides the best traction reconstruction, while simultaneously ensuring an optimum estimate of the maximum traction, at the cost of high CPU time and low efficiency. Moreover, regularization-based methods introduce the challenge of calibrating the regularization parameter, usually done under subjective criteria. On the other hand, non-regularized but constrained methods may represent a nice compromise between accuracy and efficiency, while avoiding pre-processing and calibration of such regularization parameter. It is emphasized the importance of considering not only traction reconstruction quality but also the efficiency and complexity of implementation (intrusiveness) when selecting an appropriate inverse method for TFM analysis.

A TomoSAR regularization-based method for height change detection in urban areas

Over the past years, urban change detection has always been a challenging issue for researchers. Due to the two-dimensional intrinsic property of satellite imageries and the inherent less intuitive interpretability of radar images, height change detection and SAR imagery-based change detection have been even more complicated tasks. However, TomoSAR-based height change detection is a novel topic on which only a few research works have ever been realized. This study explores the viability of tomographic-based change detection in urban areas, tackling the common challenges encountered in 2D change detection with SAR images. In contrast with the standard TomoSAR methods that usually suffer from the ill-posed inverse problem, the TomoSAR regularization-based methods are considered powerful aids for reconstructing the reflectivity function of the observed scenes. The TomoSAR regularization-based method utilizes the Constrained Least Squares (CLS) and the Weighted Constrained Least Squares (WCLS) as two efficient strategies for coming up with a well-posed solution to the prevalent ill-posed TomoSAR problem. They can assist SAR tomography to deal with the possible impairing issues arising from low numbers and the distribution of baselines. This advantage helps enhance the accuracy of the results. The experimental results are investigated in terms of completeness and correctness measures to evaluate the feasibility and reliability of the proposed method. The proposed method for height change detection has been performed on two datasets acquired by the TerraSAR-X sensor in 2013 and the Sentinel-1 sensor in 2022 over Tehran City, Iran. The final results demonstrate that the proposed algorithm can detect height changes with good feasibility and reliability in terms of completeness and correctness of better than 84% and 77%, respectively.

Efficient Spiking Neural Networks with Sparse Selective Activation for Continual Learning

The next generation of machine intelligence requires the capability of continual learning to acquire new knowledge without forgetting the old one while conserving limited computing resources. Spiking neural networks (SNNs), compared to artificial neural networks (ANNs), have more characteristics that align with biological neurons, which may be helpful as a potential gating function for knowledge maintenance in neural networks. Inspired by the selective sparse activation principle of context gating in biological systems, we present a novel SNN model with selective activation to achieve continual learning. The trace-based K-Winner-Take-All (K-WTA) and variable threshold components are designed to form the sparsity in selective activation in spatial and temporal dimensions of spiking neurons, which promotes the subpopulation of neuron activation to perform specific tasks. As a result, continual learning can be maintained by routing different tasks via different populations of neurons in the network. The experiments are conducted on MNIST and CIFAR10 datasets under the class incremental setting. The results show that the proposed SNN model achieves competitive performance similar to and even surpasses the other regularization-based methods deployed under traditional ANNs.

Interpolation-Based Contrastive Learning for Few-Label Semi-Supervised Learning.

Semi-supervised learning (SSL) has long been proved to be an effective technique to construct powerful models with limited labels. In the existing literature, consistency regularization-based methods, which force the perturbed samples to have similar predictions with the original ones have attracted much attention for their promising accuracy. However, we observe that the performance of such methods decreases drastically when the labels get extremely limited, e.g., 2 or 3 labels for each category. Our empirical study finds that the main problem lies with the drift of semantic information in the procedure of data augmentation. The problem can be alleviated when enough supervision is provided. However, when little guidance is available, the incorrect regularization would mislead the network and undermine the performance of the algorithm. To tackle the problem, we: 1) propose an interpolation-based method to construct more reliable positive sample pairs and 2) design a novel contrastive loss to guide the embedding of the learned network to change linearly between samples so as to improve the discriminative capability of the network by enlarging the margin decision boundaries. Since no destructive regularization is introduced, the performance of our proposed algorithm is largely improved. Specifically, the proposed algorithm outperforms the second best algorithm (Comatch) with 5.3% by achieving 88.73% classification accuracy when only two labels are available for each class on the CIFAR-10 dataset. Moreover, we further prove the generality of the proposed method by improving the performance of the existing state-of-the-art algorithms considerably with our proposed strategy. The corresponding code is available at https://github.com/xihongyang1999/ICL_SSL.

Time multiscale regularization for nonlinear image registration

Regularization-based methods are commonly used for image registration. However, fixed regularizers have limitations in capturing details and describing the dynamic registration process. To address this issue, we propose a time multiscale registration framework for nonlinear image registration in this paper. Our approach replaces the fixed regularizer with a monotone decreasing sequence, and iteratively uses the residual of the previous step as the input for registration. Particularly, first, we introduce a dynamically varying regularization strategy that updates regularizers at each iteration and incorporates them with a multiscale framework. This approach guarantees an overall smooth deformation field in the initial stage of registration and fine-tunes local details as the images become more similar. We then deduce convergence analysis under certain conditions on the regularizers and parameters. Further, we introduce a TV-like regularizer to demonstrate the efficiency of our method. Finally, we compare our proposed multiscale algorithm with some existing methods on both synthetic images and pulmonary computed tomography (CT) images. The experimental results validate that our proposed algorithm outperforms the compared methods, especially in preserving details during image registration with sharp structures.

Online Continual Learning in Acoustic Scene Classification: An Empirical Study.

Numerous deep learning methods for acoustic scene classification (ASC) have been proposed to improve the classification accuracy of sound events. However, only a few studies have focused on continual learning (CL) wherein a model continually learns to solve issues with task changes. Therefore, in this study, we systematically analyzed the performance of ten recent CL methods to provide guidelines regarding their performances. The CL methods included two regularization-based methods and eight replay-based methods. First, we defined realistic and difficult scenarios such as online class-incremental (OCI) and online domain-incremental (ODI) cases for three public sound datasets. Then, we systematically analyzed the performance of each CL method in terms of average accuracy, average forgetting, and training time. In OCI scenarios, iCaRL and SCR showed the best performance for small buffer sizes, and GDumb showed the best performance for large buffer sizes. In ODI scenarios, SCR adopting supervised contrastive learning consistently outperformed the other methods, regardless of the memory buffer size. Most replay-based methods have an almost constant training time, regardless of the memory buffer size, and their performance increases with an increase in the memory buffer size. Based on these results, we must first consider GDumb/SCR for the continual learning methods for ASC.

Boundary-constrained robust regularization for single image dehazing

Generally, for single image dehazing, regularization-based schemes improve the initial transmission map iteratively by using a guidance map as a structural prior. We conducted experiments on a large number of hazy images and observed that a constrained transmission map affects the quality of the recovered image. However, regularization-based methods do not constrain the transmission map to its physically valid range during the iterative process. It degrades its robustness to outliers, and consequently, deteriorates the quality of the recovered image. In addition, conventional methods fuse the structural information of the guidance and initial transmission map without considering any structural differences between them. To address these issues, in this paper, we present a robust regularization scheme that constraints the transmission map during its enhancement. In the proposed scheme, a nonconvex energy function is constructed that leverages the mutual structural information of the guidance and transmission map. The nonconvex problem is solved by a majorize-minimization algorithm, and the intermediate transmission maps are constrained through the appropriate lower and upper bounds. The retrieved transmission map has better edge-preserving properties, and ultimately, results in a high-quality haze-free image that has faithful colors and fine details. The proposed scheme is tested on benchmark datasets and results are evaluated through quantitative metrics. The comparative analysis has revealed the effectiveness of the proposed scheme.

An adaptive image restoration algorithm based on hybrid total variation regularization

In imaging systems, the mixed Poisson-Gaussian noise (MPGN) model can accurately describe the noise present. Total variation (TV) regularization-based methods have been widely utilized for Poisson-Gaussian removal with edge-preserving. However, TV regularization sometimes causes staircase artifacts with piecewise constants. To overcome this issue, we propose a new model in which the regularization term is represented by a combination of total variation and high-order total variation. We study the existence and uniqueness of the minimizer for the considered model. Numerically, the minimization problem can be efficiently solved by the alternating minimization method. Furthermore, we give rigorous convergence analyses of our algorithm. Experiments results are provided to demonstrate the superiority of our proposed hybrid model and algorithm for deblurring and denoising images simultaneously, in comparison with several state-of-the-art numerical algorithms.

Synchrotron microtomography image restoration via regularization representation and deep CNN prior

Background and objectiveSynchrotron-based X-ray microtomography (S-µCT) is a promising imaging technique that plays an important role in modern medical science. S-µCT systems often cause various artifacts and noises in the reconstructed CT images, such as ring artifacts, quantum noise, and electronic noise. In most situations, such noise and artifacts occur simultaneously, which results in a deterioration in the image quality and affects subsequent research. Due to the complexity of the distribution of these mixed artifacts and noise, it is difficult to restore the corrupted images. To address this issue, we propose a novel algorithm to remove mixed artifacts and noise from S-µCT images simultaneously. MethodsThere are two important aspects of our method. Regarding ring artifacts, because of their specific structural characteristics, regularization-based methods are more suitable; thus, low-rank tensor decomposition and total variation are utilized to represent their directional and locally piecewise smoothness properties. Moreover, to determine the implicit prior of the random noise, a convolutional neural network (CNN) based method is used. The advantages of traditional regularization and the deep CNN are then combined and embedded in a plug-and-play framework. Hence, an efficient image restoration algorithm is proposed to address the problem of mixed artifacts and noise in S-µCT images. ResultsOur proposed method was assessed by utilizing simulations and real data experiments. The qualitative results showed that the proposed method could effectively remove ring artifacts as well as random noise. The quantitative results demonstrated that the proposed method achieved almost the best results in terms of PSNR, SSIM and MAE compared to other methods. ConclusionsThe proposed method can serve as an effective tool for restoring corrupted S-µCT images, and it has the potential to promote the application of S-µCT.

Deblurring galaxy images with Tikhonov regularization on magnitude domain

Abstract We propose a regularization-based deblurring method that works efficiently for galaxy images. The spatial resolution of a ground-based telescope is generally limited by seeing conditions and is much worse than space-based telescopes. This circumstance has generated considerable research interest in the restoration of spatial resolution. Since image deblurring is a typical inverse problem and often ill-posed, solutions tend to be unstable. To obtain a stable solution, much research has adopted regularization-based methods for image deblurring, but the regularization term is not necessarily appropriate for galaxy images. Although galaxies have an exponential or Sérsic profile, the conventional regularization assumes the image profiles to behave linearly in space. The significant deviation between the assumption and real situations leads to blurring of the images and smoothing out the detailed structures. Clearly, regularization on logarithmic domain, i.e., magnitude domain, should provide a more appropriate assumption, which we explore in this study. We formulate a problem of deblurring galaxy images by an objective function with a Tikhonov regularization term on a magnitude domain. We introduce an iterative algorithm minimizing the objective function with a primal–dual splitting method. We investigate the feasibility of the proposed method using simulation and observation images. In the simulation, we blur galaxy images with a realistic point spread function and add both Gaussian and Poisson noise. For the evaluation with the observed images, we use galaxy images taken by the Subaru HSC-SSP. Both of these evaluations show that our method successfully recovers the spatial resolution of the deblurred images and significantly outperforms the conventional methods. The code is publicly available from the GitHub 〈https://github.com/kzmurata-astro/PSFdeconv_amag〉.

Multi-branch attention prior based parameterized generative adversarial network for fast and accurate limited-projection reconstruction in fluorescence molecular tomography.

Limited-projection fluorescence molecular tomography (FMT) allows rapid reconstruction of the three-dimensional (3D) distribution of fluorescent targets within a shorter data acquisition time. However, the limited-projection FMT is severely ill-posed and ill-conditioned due to insufficient fluorescence measurements and the strong scattering properties of photons in biological tissues. Previously, regularization-based methods, combined with the sparse distribution of fluorescent sources, have been commonly used to alleviate the severe ill-posed nature of the limited-projection FMT. Due to the complex iterative computations, time-consuming solution procedures, and less stable reconstruction results, the limited-projection FMT remains an intractable challenge for achieving fast and accurate reconstructions. In this work, we completely discard the previous iterative solving-based reconstruction themes and propose multi-branch attention prior based parameterized generative adversarial network (MAP-PGAN) to achieve fast and accurate limited-projection FMT reconstruction. Firstly, the multi-branch attention can provide parameterized weighted sparse prior information for fluorescent sources, enabling MAP-PGAN to effectively mitigate the ill-posedness and significantly improve the reconstruction accuracy of limited-projection FMT. Secondly, since the end-to-end direct reconstruction strategy is adopted, the complex iterative computation process in traditional regularization algorithms can be avoided, thus greatly accelerating the 3D visualization process. The numerical simulation results show that the proposed MAP-PGAN method outperforms the state-of-the-art methods in terms of localization accuracy and morphological recovery. Meanwhile, the reconstruction time is only about 0.18s, which is about 100 to 1000 times faster than the conventional iteration-based regularization algorithms. The reconstruction results from the physical phantoms and in vivo experiments further demonstrate the feasibility and practicality of the MAP-PGAN method in achieving fast and accurate limited-projection FMT reconstruction.

Multi-fidelity and learning-regularization for single image super resolution

Ill-posedness results in regularization-based methods being widely used in single image super-resolution (SISR). However, producing super-resolved images with finer details and fewer artifacts is still a great challenge. With the help of the plug-and-play framework, we introduce a novel fidelity term and learned prior knowledge to produce a powerful SISR model. In the proposed fidelity term, called multi-fidelity, the similarity between the observed data and simulated data is measured in terms of both intensity and gradient; these measurements can reflect the tendency of feature response results to degrade into multiple image layers. Based on the half quadratic splitting (HQS) method, the proposed SISR model is split into two sub-problems, which include the multi-fidelity and regularization terms, respectively. In this paper, we design a deep network, named as R&BED, to learn prior image knowledge. Compared with the manually designed regularization term, learned knowledge can preserve easily ignored features. Experimental results indicate that the subjective and objective metrics corresponding to the proposed method are better than those obtained using the comparison methods.

Learning Deep Subspace Projection Prior for Dual-Camera Compressive Hyperspectral Imaging

Coded aperture snapshot spectral imaging (CASSI) captures the 3-D hyperspectral images (HSI) in the form of 2-D coded images. The dual-camera compressive hyperspectral imaging (DCCHI) can effectively improve the reconstruction quality by adding a parallel complementary panchromatic camera. Several regularization-based methods have been proposed for dual-camera reconstruction. However, the handcrafted priors of these methods are limited in representing the complex intrinsic structure of HSI. In this letter, we propose to learn deep subspace projection prior for dual-camera compressive reconstruction. We first design a deep subspace projection prior regularized dual-camera compressive reconstruction model and minimize it with alternative optimization. Then, we unfold the optimization process into a network. Specifically, the deep subspace projection prior learning leads to features with low-rank characteristics, which could efficiently exploit the spectral correlation of HSI. The dual-camera compressive reconstruction network is learned in an end-to-end manner. Extensive experiments substantiate the performance and efficiency of other start-of-the-art algorithms.

Block-Term Tensor Decomposition Model Selection and Computation: The Bayesian Way

The so-called block-term decomposition (BTD) tensor model, especially in its rank-$(L_r,L_r,1)$ version, has been recently receiving increasing attention due to its enhanced ability of representing systems and signals that are composed of \emph{blocks} of rank higher than one, a scenario encountered in numerous and diverse applications. Uniqueness conditions and fitting methods have thus been thoroughly studied. Nevertheless, the challenging problem of estimating the BTD model structure, namely the number of block terms, $R$, and their individual ranks, $L_r$, has only recently started to attract significant attention, mainly through regularization-based approaches which entail the need to tune the regularization parameter(s). In this work, we build on ideas of sparse Bayesian learning (SBL) and put forward a fully automated Bayesian approach. Through a suitably crafted multi-level \emph{hierarchical} probabilistic model, which gives rise to heavy-tailed prior distributions for the BTD factors, structured sparsity is \emph{jointly} imposed. Ranks are then estimated from the numbers of blocks ($R$) and columns ($L_r$) of non-negligible energy. Approximate posterior inference is implemented, within the variational inference framework. The resulting iterative algorithm completely avoids hyperparameter tuning, which is a significant defect of regularization-based methods. Alternative probabilistic models are also explored and the connections with their regularization-based counterparts are brought to light with the aid of the associated maximum a-posteriori (MAP) estimators. We report simulation results with both synthetic and real-word data, which demonstrate the merits of the proposed method in terms of both rank estimation and model fitting as compared to state-of-the-art relevant methods.

Fuzzy clustering algorithms with distance metric learning and entropy regularization

Clustering has been used in various fields, such as image processing, data mining, pattern recognition, and statistical analysis. Generally, clustering algorithms consider all variables equally relevant or not correlated. Nevertheless, the pattern of data samples in the multidimensional space can be geometrically complicated, e.g., clusters may exist in different subsets of features. In this regard, new soft subspace clustering algorithms have been proposed, in which the correlation and relevance of variables are considered to improve their performance. Since regularization-based methods are robust for initializations, the approaches proposed introduce an entropy regularization term for controlling the membership degree of the objects. Such regularizations are popular due to high performance in large-scale data clustering and low computational complexity. These three-step iterative algorithms provide a fuzzy partition, a representative for each cluster, and the relevance weight of the variables or their correlation by minimizing a suitable objective function. Several experiments on synthetic and real datasets, including their application to the segmentation of noisy image textures, demonstrate the usefulness of the proposed clustering methods.

SpaceNet: Make Free Space for Continual Learning

The continual learning (CL) paradigm aims to enable neural networks to learn tasks continually in a sequential fashion. The fundamental challenge in this learning paradigm is catastrophic forgetting previously learned tasks when the model is optimized for a new task, especially when their data is not accessible. Current architectural-based methods aim at alleviating the catastrophic forgetting problem but at the expense of expanding the capacity of the model. Regularization-based methods maintain a fixed model capacity; however, previous studies showed the huge performance degradation of these methods when the task identity is not available during inference (e.g. class incremental learning scenario). In this work, we propose a novel architectural-based method referred as SpaceNet11Code available at: https://github.com/GhadaSokar/SpaceNet for class incremental learning scenario where we utilize the available fixed capacity of the model intelligently. SpaceNet trains sparse deep neural networks from scratch in an adaptive way that compresses the sparse connections of each task in a compact number of neurons. The adaptive training of the sparse connections results in sparse representations that reduce the interference between the tasks. Experimental results show the robustness of our proposed method against catastrophic forgetting old tasks and the efficiency of SpaceNet in utilizing the available capacity of the model, leaving space for more tasks to be learned. In particular, when SpaceNet is tested on the well-known benchmarks for CL: split MNIST, split Fashion-MNIST, CIFAR-10/100, and iCIFAR100, it outperforms regularization-based methods by a big performance gap. Moreover, it achieves better performance than architectural-based methods without model expansion and achieves comparable results with rehearsal-based methods, while offering a huge memory reduction.