Existing Regularization Methods Research Articles

CL-BPUWM: continuous learning with Bayesian parameter updating and weight memory

Catastrophic forgetting in neural networks is a common problem, in which neural networks lose information from previous tasks after training on new tasks. Although adopting a regularization method that preferentially retains the parameters important to the previous task to avoid catastrophic forgetting has a positive effect; existing regularization methods cause the gradient to be near zero because the loss is at the local minimum. To solve this problem, we propose a new continuous learning method with Bayesian parameter updating and weight memory (CL-BPUWM). First, a parameter updating method based on the Bayes criterion is proposed to allow the neural network to gradually obtain new knowledge. The diagonal of the Fisher information matrix is then introduced to significantly minimize computation and increase parameter updating efficiency. Second, we suggest calculating the importance weight by observing how changes in each network parameter affect the model prediction output. In the process of model parameter updating, the Fisher information matrix and the sensitivity of the network are used as the quadratic penalty terms of the loss function. Finally, we apply dropout regularization to reduce model overfitting during training and to improve model generalizability. CL-BPUWM performs very well in continuous learning for classification tasks on CIFAR-100 dataset, CIFAR-10 dataset, and MNIST dataset. On CIFAR-100 dataset, it is 0.8%, 1.03% and 0.75% higher than the best performing regularization method (EWC) in three task partitions. On CIFAR-10 dataset, it is 2.25% higher than the regularization method (EWC) and 0.7% higher than the scaled method (GR). It is 0.66% higher than the regularization method (EWC) on the MNIST dataset. When the CL-BPUWM method was combined with the brain-inspired replay model under the CIFAR-100 and CIFAR-10 datasets, the classification accuracy was 2.35% and 5.38% higher than that of the baseline method, BI-R + SI.

Integrating L1 and weighted L2 regularization for moving force identification from combined response measurements

Existing regularization methods have been proven effective in the ill-posed problem of moving force identification (MFI). However, both the constant and time-varying components of moving forces cannot be accurately identified at the same time, especially under higher measurement noises. To address this issue, a new MFI method is proposed by integrating the L1 and weighted L2 regularization methods in this study, where the presented weighted L2 regularization strategy achieves a more nuanced balance of moving force components through the incorporation of weight coefficients. Furthermore, the inherent solving characteristics of L1 and weighted L2 regularization methods are fully utilized to solve different components in moving forces separately. Numerical simulations and experimental verifications are carried out to assess the effectiveness of the proposed method. The results show that the proposed method can accurately identify moving forces even under 25% higher noise levels and it can effectively evaluate the axle loads and gross vehicle weight of the experimental vehicle. Besides, comparative studies further show that the proposed method outperforms the existing L1 and L2 regularization methods in the identification of both constant and time-varying forces, which provides a potential MFI tool for a higher accuracy and noise robustness to fix the MFI problem in practice.

Photoacoustic image reconstruction with anobjective function using TGV and ESTGV asaregularization functional.

Photoacoustic tomographic imaging is a non-invasive medical diagnostic technology for visualizing biological tissue. However, the inverse problem and noise in photoacoustic signals often cause blurred images. Existing regularization methods struggle with staircasing artifacts and edge preservation. To overcome this, an objective function incorporating total generalized variation (TGV) is proposed. However, it failed with high-density Gaussian noise. To address this, an extended version called edge-guided second-order TGV (ESTGV) is introduced. For sparsification, wavelet transform and discrete cosine transform are introduced, while the fast-composite-splitting algorithm is employed for the inverse problem solution. Experimental validation demonstrates the potential of these approaches.

Compressed lightweight deep learning models for resource‐constrained Internet of things devices in the healthcare sector

AbstractThe performance of convolutional neural networks (CNNs) in image classification and object detection has been remarkable, even though they contain millions and billions of parameters. This over‐parameterization of CNN makes them both memory‐intensive and computationally complex and exhaustive. This greatly hinders the application of CNNs in resource‐constrained environments such as Internet of things (IoT) and edge devices. This poses a critical challenge for CNNs in deploying these powerful computer vision tools to mobile devices, which needs immediate attention. In this study, we have proposed a novel technique based on non‐convex optimization, max‐norm regularization. The max‐norm will structurally prune the number of parameters without compromising the model's performance. The proximal gradient descent algorithm is used for network optimization while using this non‐convex regularizer. The max‐norm is combined with the channel pruning to achieve more sparse CNN networks. Later, the pruned network can be easily deployed in the resource‐constrained application environment. The proposed technique is tested on several benchmark datasets for validation. In addition, in this study, the sparsified CNNs are used for biomedical image analysis using the BRAIN MRI dataset. This sparsely trained CNN model can later serve as the best lightweight model applicable in the IoT healthcare sector for detecting and classifying three types of brain tumours, one of the most life‐threatening diseases whose early detection can save the costly lives of human beings. This is the first paper to propose the novel max‐norm regularizer to enforce sparse learning through CNNs. The paper provides a detailed analysis of convex and non‐convex regularizers before presenting the proposed novel max‐norm regularizer. Finally, the paper compares the proposed max‐norm regularizer with existing regularization methods using state‐of‐the‐art CNN models.

SCALAR ON NETWORK REGRESSION VIA BOOSTING.

Neuroimaging studies have a growing interest in learning the association between the individual brain connectivity networks and their clinical characteristics. It is also of great interest to identify the sub brain networks as biomarkers to predict the clinical symptoms, such as disease status, potentially providing insight on neuropathology. This motivates the need for developing a new type of regression model where the response variable is scalar, and predictors are networks that are typically represented as adjacent matrices or weighted adjacent matrices, to which we refer as scalar-on-network regression. In this work, we develop a new boosting method for model fitting with sub-network markers selection. Our approach, as opposed to group lasso or other existing regularization methods, is essentially a gradient descent algorithm leveraging known network structure. We demonstrate the utility of our methods via simulation studies and analysis of the resting-state fMRI data in a cognitive developmental cohort study.

A Recursive Hull and Signal-Based Building Footprint Generation from Airborne LiDAR Data

Automatically generating a building footprint from an airborne LiDAR point cloud is an active research topic because of its widespread usage in numerous applications. This paper presents an efficient and automated workflow for generating building footprints from pre-classified LiDAR data. In this workflow, LiDAR points that belong to the building category are first segmented into multiple clusters by applying the grid-based DBSCAN clustering algorithm. Each cluster contains the points of an individual building. Then, the outermost points of each building are extracted, on which the recursive convex hull algorithm is applied to generate the initial outline of each building. Since LiDAR points are irregularly distributed, the initial building outline contains irregular zig-zag shapes. In order to achieve a regularized building footprint that is close to the true building boundary, a signal-based regularization algorithm is developed. The initial outline is first transformed into a signal, which can reveal the wholistic geometric structure of the building outline after applying a denoising procedure. By analyzing the denoised signal, the locations of corners are identified, and the regularized building footprint is generated. The performance of the proposed workflow is tested and evaluated using two datasets that have different point densities and building types. The qualitative assessment reveals that the proposed workflow has a satisfying performance in generating building footprints even for building with complex structures. The quantitative assessment compares the performance of signal-based regularization with existing regularization methods using the 149 buildings contained in the test dataset. The experimental result shows the proposed method has achieved superior results based on a number of commonly used accuracy metrics.

Feature variance regularization method for autoencoder-based one-class classification

One-class classification (OCC) has been being used in various research fields, since it is able to design classifiers using the data from a single class. Among various methods for OCC, principal component analysis (PCA) is one of the most widely used ones, whose nonlinear extension can be performed by autoencoders. Although the existing regularization methods, such as L1 and L2 regularizations, can prevent the total variance of autoencoder's bottleneck layer from exploding, they cannot effectively reduce it below certain levels to alleviate the problem of variance inflation. To this end, in this work, a novel variance regularization method is proposed, which directly controls the total variance of the bottleneck layer. Case studies are carried out using two datasets (MNIST dataset and Tennessee Eastman dataset) to illustrate its ability as a regularizer, and as an enhancer for the design of one-class classifiers (in terms of performance and training time).

Toward durable representations for continual learning

Continual learning models are known to suffer from catastrophic forgetting. Existing regularization methods to countering forgetting operate by penalizing large changes to learned parameters. A significant downside to these methods, however, is that, by effectively freezing model parameters, they gradually suspend the capacity of a model to learn new tasks. In this paper, we explore an alternative approach to the continual learning problem that aims to circumvent this downside. In particular, we ask the question: instead of forcing continual learning models to remember the past, can we modify the learning process from the start, such that the learned representations are less susceptible to forgetting? To this end, we explore multiple methods that could potentially encourage durable representations. We demonstrate empirically that the use of unsupervised auxiliary tasks achieves significant reduction in parameter re-optimization across tasks, and consequently reduces forgetting, without explicitly penalizing forgetting. Moreover, we propose a distance metric to track internal model dynamics across tasks, and use it to gain insight into the workings of our proposed approach, as well as other recently proposed methods.

Adaptive Knowledge Driven Regularization for Deep Neural Networks

In many real-world applications, the amount of data available for training is often limited, and thus inductive bias and auxiliary knowledge are much needed for regularizing model training. One popular regularization method is to impose prior distribution assumptions on model parameters, and many recent works also attempt to regularize training by integrating external knowledge into specific neurons. However, existing regularization methods did not take account of the interaction between connected neuron pairs, which is invaluable internal knowledge for adaptive regularization for better representation learning as training progresses. In this paper, we explicitly take into account the interaction between connected neurons, and propose an adaptive internal knowledge driven regularization method, CORR-Reg. The key idea of CORR-Reg is to give a higher significance weight to connections of more correlated neuron pairs. The significance weights adaptively identify more important input neurons for each neuron. Instead of regularizing connection model parameters with a static strength such as weight decay, CORR-Reg imposes weaker regularization strength on more significant connections. As a consequence, neurons attend to more informative input features and thus learn more diversified and discriminative representation. We derive CORR-Reg with Bayesian inference framework and propose a novel optimization algorithm with Lagrange multiplier method and Stochastic Gradient Descent. Extensive evaluations on diverse benchmark datasets and neural network structures show that CORR-Reg achieves significant improvement over state-of-the-art regularization methods.

A Joint Framework for Seismic Signal Denoising Using Total Generalized Variation and Shearlet Transform

Seismic exploration is a remote-sensing tool applied in a great many projects for engineering and resource-exploration purposes. Random noise suppression is one of the key steps in seismic-signal processing, especially those with important details and features. The threshold-shrinkage method based on Shearlet transform has been effectively applied in seismic-signal denoising. However, the method usually introduces the boundary effect, which influences the imaging quality. The denoising method of total generalized variation (TGV) is easy to produce ‘oil painting’ effect, but it can effectively suppress the boundary effect. This paper proposes a denoising method based on Shearlet threshold-shrinkage and TGV for making full use of their characteristics, which can recover both edges and fine details much better than the existing regularization methods. First, we use the Shearlet threshold-shrinkage result as the input of TGV to obtain the primary denoising result and the residual profile. Second, we use the interactive iteration of Shearlet threshold-shrinkage and TGV to extract the signals efficiently from the residual profile and perform the effective signals stack continuously. During the processing, the adaptive-weight factor is combined for estimating the optimal denoising result. Last, the final estimated denoising result is obtained when the stopping criterion is met or the maximum number of iterations is reached. The synthetic and field results show that the proposed method can effectively suppress random noise. In addition, it can also remove the boundary effect and ‘oil painting’ effect, which further improves the signal-to-noise ratio (SNR).

Consistent Fixed-Effects Selection in Ultra-high dimensional Linear Mixed Models with Error-Covariate Endogeneity

Recently, applied sciences, including longitudinal and clustered studies in biomedicine require the analysis of ultra-high dimensional linear mixed effects models where we need to select important fixed effect variables from a vast pool of available candidates. However, all existing literature assume that all the available covariates and random effect components are independent of the model error which is often violated (endogeneity) in practice. In this paper, we first investigate this important issue in ultra-high dimensional linear mixed effects models with particular focus on the fixed effects selection. We study the effects of different types of endogeneity on existing regularization methods and prove their inconsistencies. Then, we propose a new profiled focused generalized method of moments (PFGMM) approach to consistently select fixed effects under 'error-covariate' endogeneity, i.e., in the presence of correlation between the model error and covariates. Our proposal is proved to be oracle consistent with probability tending to one and works well under most other type of endogeneity too. Additionally, we also propose and illustrate a few consistent parameter estimators, including those of the variance components, along with variable selection through PFGMM. Empirical simulations and an interesting real data example further support the claimed utility of our proposal.

Nonsparse Learning with Latent Variables

A New Nonsparse Learning Methodology for High-Dimensional Data Analysis Is Coming

Weighted Nuclear Norm Minimization-Based Regularization Method for Image Restoration

Regularization methods have been substantially applied in image restoration due to the ill-posedness of the image restoration problem. Different assumptions or priors on images are applied in the construction of image regularization methods. In recent years, matrix low-rank approximation has been successfully introduced in the image denoising problem and significant denoising effects have been achieved. Low-rank matrix minimization is an NP-hard problem and it is often replaced with the matrix’s weighted nuclear norm minimization (WNNM). The assumption that an image contains an extensive amount of self-similarity is the basis for the construction of the matrix low-rank approximation-based image denoising method. In this paper, we develop a model for image restoration using the sum of block matching matrices’ weighted nuclear norm to be the regularization term in the cost function. An alternating iterative algorithm is designed to solve the proposed model and the convergence analyses of the algorithm are also presented. Numerical experiments show that the proposed method can recover the images much better than the existing regularization methods in terms of both recovered quantities and visual qualities.

A novel reconstruction approach combining OSEM and split Bregman method for low dose CT

PurposeLow dose CT imaging is an important research hotspot in the field of medical imaging. On the condition of low dose scanning, the commonly used filtered back projection (FBP) algorithm in the case of normal dose cannot meet the requirements with low signal-to-noise ratio (SNR), stripe artifacts and other problems. The algorithms of statistical iteration type can better handle low dose projection data. Existing regularization methods have been shown to deal with this problem to a large extent. Because their regular items are fixed, their adaptability to low dose conditions is not well. The main purpose of this paper is to explore the new method to improve the quality of CT reconstruction image at low dose condition. MethodsA novel approach is proposed based on OSEM and split Bregman method (OSEM-SBTV) for low dose CT. It includes two steps: OSEM solving image reconstruction and split Bregman method solving total variation denoising. ResultsCompared with OSEM, results show that OSEM-SBTV has better performance in suppressing noise and smoothing than the classical OSEM. For comparison of profiles of Tikhonov, L1 and TV regularization models, the results of L1 norm are most affected by noise, and the profiles fluctuate greatly. The profile results of Tikhonov and TV norm are over smooth, which results in no representation of the profile information of the middle circle of the image at all in the middle of the profile. ConclusionsThe proposed approach can keep the reconstructed image smooth while maintaining the fine structure. This is a good approach to deal with low dose CT image reconstruction.

CRLEDD: Regularized Causalities Learning for Early Detection of Diseases Using Electronic Health Record (EHR) Data

The availability of Electronic Health Records (EHR) in health care settings has provided tremendous opportunities for early disease detection. While many supervised learning models have been adopted for EHR-based disease early detection, the ill-posed inverse problem in the parameter learning has imposed a significant challenge on improving the accuracy of these algorithms. In this paper, we propose CRLEDD - Causality-Regularized Learning for Early Detection of Disease, an algorithm to improve the performance of Linear Discriminant Analysis (LDA) on top of diagnosis-frequency vector data representation. While most existing regularization methods exploit sparsity regularization to improve detection performance, CRLEDD provides a unique perspective by ensuring positive semi-definiteness of the sparsified precision matrix used in LDA which is different from the regular regularization method (e.g., L2 regularization). To achieve this goal, CRLEDD employs Graphical Lasso to estimate the precision matrix in the ill-posed settings for enhanced accuracy of LDA classifiers. We perform extensive evaluation of CRLEDD using a large-scale real-world EHR dataset to predict mental health disorders (e.g., depression and anxiety) of college students from 10 universities in the U.S. We compare CRLEDD with other regularized LDA and downstream classifiers. The result shows that CRLEDD outperforms all baselines in terms of accuracy and F1 scores.

Sparse Logistic Regression With L 1/2 Penalty for Emotion Recognition in Electroencephalography Classification.

Emotion recognition based on electroencephalography (EEG) signals is a current focus in brain-computer interface research. However, the classification of EEG is difficult owing to large amounts of data and high levels of noise. Therefore, it is important to determine how to effectively extract features that include important information. Regularization, one of the effective methods for EEG signal processing, can effectively extract important features from the signal and has potential applications in EEG emotion recognition. Currently, the most popular regularization technique is Lasso (L1) and Ridge Regression (L2). In recent years, researchers have proposed many other regularization terms. In theory, Lq-type regularization has a lower q value, which means that it can be used to find solutions with better sparsity. L1/2 regularization is of Lq type (0 < q < 1) and has been shown to have many attractive properties. In this work, we studied the L1/2 penalty in sparse logistic regression for three-classification EEG emotion recognition, and used a coordinate descent algorithm and a univariate semi-threshold operator to implement L1/2 penalty logistic regression. The experimental results on simulation and real data demonstrate that our proposed method is better than other existing regularization methods. Sparse logistic regression with L1/2 penalty achieves higher classification accuracy than the conventional L1, Ridge Regression, and Elastic Net regularization methods, using fewer but more informative EEG signals. This is very important for high-dimensional small-sample EEG data and can help researchers to reduce computational complexity and improve computational accuracy. Therefore, we propose that sparse logistic regression with the L1/2 penalty is an effective technique for emotion recognition in practical classification problems.

Regularization of Building Roof Boundaries from Airborne LiDAR Data Using an Iterative CD-Spline

Building boundaries play an essential role in many applications such as urban planning and production of 3D realistic views. In this context, airborne LiDAR data have been explored for the generation of digital building models. Despite the many developed strategies, there is no method capable of encompassing all the complexities in an urban environment. In general, the vast majority of existing regularization methods are based on building boundaries that are made up of straight lines. Therefore, the development of a strategy able to model building boundaries, regardless of their degree of complexity is of high importance. To overcome the limitations of existing strategies, an iterative CD-spline (changeable degree spline) regularization method is proposed. The main contribution is the automated selection of the polynomial function that best models each segment of the building roof boundaries. Conducted experiments with real data verified the ability of the proposed approach in modeling boundaries with different levels of complexities, including buildings composed of complex curved segments and point cloud with different densities, presenting Fscore and PoLiS around 95% and 0.30 m, respectively.

Adversarial Transformations for Semi-Supervised Learning

We propose a Regularization framework based on Adversarial Transformations (RAT) for semi-supervised learning. RAT is designed to enhance robustness of the output distribution of class prediction for a given data against input perturbation. RAT is an extension of Virtual Adversarial Training (VAT) in such a way that RAT adversraialy transforms data along the underlying data distribution by a rich set of data transformation functions that leave class label invariant, whereas VAT simply produces adversarial additive noises. In addition, we verified that a technique of gradually increasing of perturbation region further improves the robustness. In experiments, we show that RAT significantly improves classification performance on CIFAR-10 and SVHN compared to existing regularization methods under standard semi-supervised image classification settings.

Matrix regularization-based method for large-scale inverse problem of force identification

Identification of force via structural responses has been widely studied due to the force is always difficult or even impossible to be measured directly. Force identification is a typical ill-posed problem. To overcome this drawback, regularization methods have been widely studied. However, existing regularization methods used for force identification belong to the vector-based method, in which unknown force and structural responses are organized in two vectors, respectively. This characteristic decides that the identified process of a long-duration problem will be time-consuming because the size of system matrix is large. In view of this, a novel matrix regularization-based method is proposed for force identification in this paper. Combing with moving time windows, the structural responses are extracted and organized in a form of matrix. A system matrix is formulated by considering both unknown force and unknown initial conditions. Then a governing equation is established, in which structural excitation sources such as force and initial condition are orderly stored in a form of matrix. To obtain a stable solution, matrix regularization is introduced for improving the matrix-based governing equation. Herein, the sparse penalty term is considered as the sum of absolute values of elements in the matrix of excitation sources. Fast iterative shrinkage-thresholding algorithm (FISTA) is applied for solving the matrix regularization model. The regularization parameter is selected according to Bayesian information criterion (BIC). Finally, numerical simulations and experimental studies are carried out for assessing the feasibility and effectiveness of the proposed method. Illustrated results show that the proposed method is effective and time-saving. It can use a system matrix with a relatively small size to dealing with a large-scale problem of force identification. Some related issues are discussed as well.

Image Restoration by Combined Order Regularization with Optimal Spatial Adaptation.

Total Variation (TV) and related extensions have been popular in image restoration due to their robust performance and wide applicability. While the original formulation is still relevant after two decades of extensive research, its extensions that combine derivatives of first and second orders are now being explored for better performance, with examples being Combined Order TV (COTV) and Total Generalized Variation (TGV). As an improvement over such multi-order convex formulations, we propose a novel non-convex regularization functional which adaptively combines Hessian-Schatten (HS) norm and first order TV (TV1) functionals with spatially varying weight. This adaptive weight itself is controlled by another regularization term; the total cost becomes the sum of this adaptively weighted HS-TV1 term, the regularization term for the adaptive weight, and the data-fitting term. The reconstruction is obtained by jointly minimizing w.r.t. the required image and the adaptive weight. We construct a block coordinate descent method for this minimization with proof of convergence, which alternates between minimization w.r.t. the required image and the adaptive weights. We derive exact computational formula for minimization w.r.t. the adaptive weight, and construct an ADMM algorithm for minimization w.r.t. to the required image. We compare the proposed method with existing regularization methods, and a recently proposed Deep GAN method using image recovery examples including MRI reconstruction and microscopy deconvolution.