Proposed Regularization Technique Research Articles

Deflated Restarting of Exponential Integrator Method with an Implicit Regularization for Efficient Transient Circuit Simulation

Exponential integrator (EI) method based on Krylov subspace approximation is a promising method for large-scale transient circuit simulation. However, it suffers from the singularity problem and consumes large subspace dimensions for stiff circuits when using the ordinary Krylov subspace. Restarting schemes are commonly applied to reduce the subspace dimension, but they also slow down the convergence and degrade the overall computational efficiency. In this paper, we first devise an implicit and sparsity-preserving regularization technique to tackle the singularity problem facing EI in the ordinary Krylov subspace framework. Next, we analyze the root cause of the slow convergence of the ordinary Krylov subspace methods when applied to stiff circuits. Based on the analysis, we propose a deflated restarting scheme, compatible with the above regularization technique, to accelerate the convergence of restarted Krylov subspace approximation for EI methods. Numerical experiments demonstrate the effectiveness of the proposed regularization technique, and up to 50% convergence improvements for Krylov subspace approximation compared to the non-deflated version.

Training Variational Networks With Multidomain Simulations: Speed-of-Sound Image Reconstruction.

Speed-of-sound (SoS) has been shown as a potential biomarker for breast cancer imaging, successfully differentiating malignant tumors from benign ones. SoS images can be reconstructed from time-of-flight measurements from ultrasound images acquired using conventional handheld ultrasound transducers. Variational networks (VNs) have recently been shown to be a potential learning-based approach for optimizing inverse problems in image reconstruction. Despite earlier promising results, these methods, however, do not generalize well from simulated to acquired data, due to the domain shift. In this work, we present for the first time a VN solution for a pulse-echo SoS image reconstruction problem using diverging waves with conventional transducers and single-sided tissue access. This is made possible by incorporating simulations with varying complexity into training. We use loop unrolling of gradient descent with momentum, with an exponentially weighted loss of outputs at each unrolled iteration in order to regularize the training. We learn norms as activation functions regularized to have smooth forms for robustness to input distribution variations. We evaluate reconstruction quality on the ray-based and full-wave simulations as well as on the tissue-mimicking phantom data, in comparison with a classical iterative [limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS)] optimization of this image reconstruction problem. We show that the proposed regularization techniques combined with multisource domain training yield substantial improvements in the domain adaptation capabilities of VN, reducing the median root mean squared error (RMSE) by 54% on a wave-based simulation data set compared to the baseline VN. We also show that on data acquired from a tissue-mimicking breast phantom, the proposed VN provides improved reconstruction in 12 ms.

Differentiating Regularization Weights -- A Simple Mechanism to Alleviate Cold Start in Recommender Systems

Matrix factorization (MF) and its extended methodologies have been studied extensively in the community of recommender systems in the last decade. Essentially, MF attempts to search for low-ranked matrices that can (1) best approximate the known rating scores, and (2) maintain low Frobenius norm for the low-ranked matrices to prevent overfitting. Since the two objectives conflict with each other, the common practice is to assign the relative importance weights as the hyper-parameters to these objectives. The two low-ranked matrices returned by MF are often interpreted as the latent factors of a user and the latent factors of an item that would affect the rating of the user on the item. As a result, it is typical that, in the loss function, we assign a regularization weight λ p on the norms of the latent factors for all users, and another regularization weight λ q on the norms of the latent factors for all the items. We argue that such a methodology probably over-simplifies the scenario. Alternatively, we probably should assign lower constraints to the latent factors associated with the items or users that reveal more information, and set higher constraints to the others. In this article, we systematically study this topic. We found that such a simple technique can improve the prediction results of the MF-based approaches based on several public datasets. Specifically, we applied the proposed methodology on three baseline models -- SVD, SVD++, and the NMF models. We found that this technique improves the prediction accuracy for all these baseline models. Perhaps more importantly, this technique better predicts the ratings on the long-tail items, i.e., the items that were rated/viewed/purchased by few users. This suggests that this approach may partially remedy the cold-start issue. The proposed method is very general and can be easily applied on various recommendation models, such as Factorization Machines, Field-aware Factorization Machines, Factorizing Personalized Markov Chains, Prod2Vec, Behavior2Vec, and so on. We release the code for reproducibility. We implemented a Python package that integrates the proposed regularization technique with the SVD, SVD++, and the NMF model. The package can be accessed at https://github.com/ncu-dart/rdf.

Non-commutative discretize-then-optimize algorithms for elliptic PDE-constrained optimal control problems

In this paper, we analyze the convergence of several optimize-then-discretize and discretize-then-optimize algorithms, based on either a second-order or a fourth-order finite difference discretization, for solving elliptic PDE-constrained optimal control problems. To ensure the convergence of a discretize-then-optimize algorithm, one well-accepted criterion is to design the discretization scheme such that the resulting discretize-then-optimize algorithm commutes with the corresponding optimize-then-discretize algorithm. In other words, both algorithms should give rise to exactly the same discrete optimality system. However, such a restrictive criterion is not trivial to fulfill. By investigating a distributed control problem governed by an elliptic equation, we first show that enforcing such a stringent condition of commutative property is only sufficient but not necessary for achieving the desired convergence. We then introduce some suitable H 1 semi-norm penalty/regularization terms to recover the lost convergence due to the inconsistency caused by the loss of commutativity. Numerical experiments are carried out to verify our theoretical analysis and also validate the effectiveness of our proposed regularization techniques.

On regularization and error estimates for the backward heat conduction problem with time-dependent thermal diffusivity factor

This paper is concerned with a backward heat conduction problem with time-dependent thermal diffusivity factor in an infinite “strip”. This problem is drastically ill-posed which is caused by the amplified infinitely growth in the frequency components. A new regularization method based on the Meyer wavelet technique is developed to solve the considered problem. Using the Meyer wavelet technique, some new stable estimates are proposed in the Hölder and Logarithmic types which are optimal in the sense of given by Tautenhahn. The stability and convergence rate of the proposed regularization technique are proved. The good performance and the high-accuracy of this technique is demonstrated through various one and two dimensional examples. Numerical simulations and some comparative results are presented.

Regularized optimization with spatial coupling for robust decision making

In high-dimensional optimization problems where a large number of decisions need to be made, the resulting solution may exhibit high sensitivity to perturbations in the input parameters, which hinders reliable decision-making. This is particularly prevalent when there is dependence between decision variables. This paper introduces a regularized optimization approach to control the trade-off between optimality and sensitivity of the solution to optimization problems that match supply and demand over a geographical area. The proposed regularization technique achieves spatial smoothing of the solution over the geographic area. It was motivated by the need of modeling intrinsic spatial dependencies between decision variables (called herein spatial coupling), thus resulting in a more realistic solution. We demonstrate the applicability of the proposed approach for multiple optimization problems. We illustrate the proposed approach using a specific application in health care access measurement, in which a smooth solution that is robust to perturbations of model parameter leads to reliable decision-making. The experimental results show that the proposed approach can be used to find a smooth and robust solution while sacrificing its optimality at a minimum level.

S-SMOOTH: Sparsity and Smoothness Enhanced EEG Brain Tomography.

EEG source imaging enables us to reconstruct current density in the brain from the electrical measurements with excellent temporal resolution (~ ms). The corresponding EEG inverse problem is an ill-posed one that has infinitely many solutions. This is due to the fact that the number of EEG sensors is usually much smaller than that of the potential dipole locations, as well as noise contamination in the recorded signals. To obtain a unique solution, regularizations can be incorporated to impose additional constraints on the solution. An appropriate choice of regularization is critically important for the reconstruction accuracy of a brain image. In this paper, we propose a novel Sparsity and SMOOthness enhanced brain TomograpHy (s-SMOOTH) method to improve the reconstruction accuracy by integrating two recently proposed regularization techniques: Total Generalized Variation (TGV) regularization and ℓ1−2 regularization. TGV is able to preserve the source edge and recover the spatial distribution of the source intensity with high accuracy. Compared to the relevant total variation (TV) regularization, TGV enhances the smoothness of the image and reduces staircasing artifacts. The traditional TGV defined on a 2D image has been widely used in the image processing field. In order to handle 3D EEG source images, we propose a voxel-based Total Generalized Variation (vTGV) regularization that extends the definition of second-order TGV from 2D planar images to 3D irregular surfaces such as cortex surface. In addition, the ℓ1−2 regularization is utilized to promote sparsity on the current density itself. We demonstrate that ℓ1−2 regularization is able to enhance sparsity and accelerate computations than ℓ1 regularization. The proposed model is solved by an efficient and robust algorithm based on the difference of convex functions algorithm (DCA) and the alternating direction method of multipliers (ADMM). Numerical experiments using synthetic data demonstrate the advantages of the proposed method over other state-of-the-art methods in terms of total reconstruction accuracy, localization accuracy and focalization degree. The application to the source localization of event-related potential data further demonstrates the performance of the proposed method in real-world scenarios.

Data-driven estimation in equilibrium using inverse optimization

Equilibrium modeling is common in a variety of fields such as game theory and transportation science. The inputs for these models, however, are often difficult to estimate, while their outputs, i.e., the equilibria they are meant to describe, are often directly observable. By combining ideas from inverse optimization with the theory of variational inequalities, we develop an efficient, data-driven technique for estimating the parameters of these models from observed equilibria. We use this technique to estimate the utility functions of players in a game from their observed actions and to estimate the congestion function on a road network from traffic count data. A distinguishing feature of our approach is that it supports both parametric and nonparametric estimation by leveraging ideas from statistical learning (kernel methods and regularization operators). In computational experiments involving Nash and Wardrop equilibria in a nonparametric setting, we find that a) we effectively estimate the unknown demand or congestion function, respectively, and b) our proposed regularization technique substantially improves the out-of-sample performance of our estimators.

Smoothly Clipped Absolute Deviation (SCAD) regularization for compressed sensing MRI Using an augmented Lagrangian scheme

PurposeCompressed sensing (CS) provides a promising framework for MR image reconstruction from highly undersampled data, thus reducing data acquisition time. In this context, sparsity-promoting regularization techniques exploit the prior knowledge that MR images are sparse or compressible in a given transform domain. In this work, a new regularization technique was introduced by iterative linearization of the non-convex smoothly clipped absolute deviation (SCAD) norm with the aim of reducing the sampling rate even lower than it is required by the conventional l1 norm while approaching an l0 norm. Materials and MethodsThe CS-MR image reconstruction was formulated as an equality-constrained optimization problem using a variable splitting technique and solved using an augmented Lagrangian (AL) method developed to accelerate the optimization of constrained problems. The performance of the resulting SCAD-based algorithm was evaluated for discrete gradients and wavelet sparsifying transforms and compared with its l1-based counterpart using phantom and clinical studies. The k-spaces of the datasets were retrospectively undersampled using different sampling trajectories. In the AL framework, the CS-MRI problem was decomposed into two simpler sub-problems, wherein the linearization of the SCAD norm resulted in an adaptively weighted soft thresholding rule with a sparsity enhancing effect. ResultsIt was demonstrated that the proposed regularization technique adaptively assigns lower weights on the thresholding of gradient fields and wavelet coefficients, and as such, is more efficient in reducing aliasing artifacts arising from k-space undersampling, when compared to its l1-based counterpart. ConclusionThe SCAD regularization improves the performance of l1-based regularization technique, especially at reduced sampling rates, and thus might be a good candidate for some applications in CS-MRI.

Model‐based optoacoustic inversions with incomplete projection data

Optoacoustic imaging is an emerging noninvasive imaging modality that can resolve optical contrast through several millimeters to centimeters of tissue with diffraction-limited resolution of ultrasound. Yet, quantified reconstruction of tissue absorption maps requires optoacoustic signals to be collected from as many locations around the object as possible. In many tomographic imaging scenarios, however, only limited-view or partial projection data are available, which has been shown to generate image artifacts and overall loss of quantification accuracy. In this article, the recently introduced interpolated-matrix-model optoacoustic inversion method is tested in different limited-view scenarios and compared to the standard backprojection algorithm. Both direct (TGSVD) and iterative (PLSQR) regularization methods are proposed to improve the accuracy of image reconstructions with their performance tested on simulated and experimental data. While for full-view tomographic data the model-based inversion has been generally shown to attain higher reconstruction accuracy compared to backprojection algorithms, the incomplete tomographic datasets lead to ill-conditioned forward matrices and, consequently, to error-prone inversions, with strong artifacts following a distinct ripple-type pattern. The proposed regularization techniques are shown to stabilize the inversion and eliminate the artifacts. Overall, it has been determined that the regularized interpolated-matrix-model-based optoacoustic inversions show higher accuracy than reconstructions with the standard backprojection algorithm. Finally, the combination of model-based inversion with PLSQR or TGSVD regularization methods can lead to an accurate reconstruction of limited-projection angle optoacoustic data and practical systems for optoacoustic imaging in many realistic cases where the full-view dataset is unavailable.

Dealing with Zero Flows in Solving the Nonlinear Equations for Water Distribution Systems

Three issues concerning the iterative solution of the nonlinear equations governing the flows and heads in a water distribution system network are considered. Zero flows cause a computation failure (division by zero) when the Global Gradient Algorithm of Todini and Pilati is used to solve for the steady state of a system in which the head loss is modeled by the Hazen-Williams formula. The proposed regularization technique overcomes this failure as a solution to this first issue. The second issue relates to zero flows in the Darcy-Weisbach formulation. This work explains for the first time why zero flows do not lead to a division by zero where the head loss is modeled by the Darcy-Weisbach formula. In this paper, the authors show how to handle the computation appropriately in the case of laminar flow (the only instance in which zero flows may occur). However, as is shown, a significant loss of accuracy can result if the Jacobian matrix, necessary for the solution process, becomes poorly conditioned, and so it is recommended that the regularization technique be used for the Darcy-Weisbach case also. Only a modest extra computational cost is incurred when the technique is applied. The third issue relates to a new convergence stopping criterion for the iterative process based on the infinity-norm of the vector of nodal head differences between one iteration and the next. This test is recommended because it has a more natural physical interpretation than the relative discharge stopping criterion that is currently used in standard software packages such as EPANET. In addition, it is recommended to check the infinity norms of the residuals once iteration has been stopped. The residuals test ensures that inaccurate solutions are not accepted.

A pre-regularization scheme for the reconstruction of a spatial dependent scattering albedo using a hybrid ant colony optimization implementation

This work presents a regularization technique applied to an inverse radiative transfer problem formulated as a finite dimensional optimization problem and solved by a hybridization of the ant colony optimization (ACO) with the Levenberg–Marquardt method. It is considered a one-dimensional isotropically-scattering medium with finite optical thickness, space dependent scattering albedo and plane-parallel geometry. The direct radiative transfer problem models transmission of radiation through this medium by the linear version of the Boltzmann equation considering polar angle discretization and azimuthal symmetry. A discrete ordinates method combined with the finite difference method is employed to solve it. Reconstruction of the albedo profile is performed from the intensities of the polar-discretized emergent radiation acquired with external detectors, using a recently proposed regularization technique. Since smooth albedo profiles are expected, such information is used in a new generation of ants in order to perform a pre-selection of the ants. This scheme can be viewed as a kind of pre-regularization in face of the reconstructed profiles that are smooth and show good agreement with the exact solution. In addition, this scheme saves processing time as fewer candidate solutions (ants) are evaluated. Noiseless and noisy data of the emergent radiation intensities were employed in the reconstructions.

A regularized Lagrangian finite point method for the simulation of incompressible viscous flows

In this paper we present a regularized Lagrangian finite point method (RLFPM) for the numerical simulation of incompressible viscous flows. A Lagrangian finite point scheme is applied to the projection method for the incompressible Navier–Stokes equations. The approximation of spatial derivatives is obtained by the weighted least squares method. The pressure Poisson equation with Neumann boundary condition is solved by a stabilized finite point method. A key aspect of the present approach is the periodic redistribution of the particle locations, which are being distorted by the flow. Again, weighted least squares approximation is implemented to interpolate the properties of the old particles onto the new particle locations. With the proposed regularization technique, problems associated with the flow-induced irregularity of particle distribution in the Lagrangian finite point scheme are circumvented. Three numerical examples, Taylor–Green flow, lid-driven flow in a cavity and flow through a periodic lattice of cylinders, are presented to validate the proposed methodology. The problem of extra diffusion caused by regularization is discussed. The results demonstrate that RLFPM is able to perform accurate and stable simulations of incompressible viscous flows.

Dynamic regularization of input estimation problems by explicit block inversion

Input estimation problems in structural dynamics often lead to solving damped least squares problems. Explicit block inversion algorithms are developed to invert the associated upper block triangular Toeplitz matrix. Moreover, the derived inversion algorithms enable the introduction of a new type of sequential regularization technique. For optimal regularization parameters, it is demonstrated by a numerical example that the solution using the proposed regularization technique has an upper bound equal to zeroth-order Tikhonov regularization.

Localization Issues in Force-Based Frame Elements

This paper summarizes the results of an investigation on objective modeling and prediction of deformation localization in nonlinear force-based frame elements. Emphasis is placed on the behavior of reinforced concrete structures where localization is most likely to occur. The discussion begins with a brief review of the force-based element formulation. Next, plastic hinge behavior is classified into three categories: hardening plastic hinges, perfectly plastic hinges, and softening plastic hinges. The three types of inelastic behavior pose an increasing challenge to objective postpeak modeling. The physical and numerical characteristics of localized failure are outlined. Two regularization techniques for maintaining postpeak objectivity are proposed: (1) a constant fracture energy criterion, which provides objective response on the element force-displacement level; and (2) geometric scaling, which provides objective response on the local moment-curvature level in plastic hinge regions. The discussion concludes with three applications to different reinforced concrete structures: a frame, a bridge pier, and an overreinforced concrete beam. In all three cases, the proposed regularization technique successfully yields objective model predictions.

Analytical study and numerical experiments for Laplace equation with overspecified boundary conditions

In this paper, the window function, e − αk 2 , is applied to regularize the divergent problem which occurs in the Laplace equation with overspecified boundary conditions in an infinite strip region. To deal with this ill-posed problem, the corner of the L-curve is chosen as the compromise point to determine the optimal α of the Gaussian window, e − αk 2 , so that the high wave-number ( k) content can be suppressed instead of engineering judgement using the concept of a cutoff wave-number. From the examples shown, it is found that a reasonable solution of the unknown boundary potential can be reconstructed, and that both high wave-number content and divergent results can be avoided by using the proposed regularization technique.

Application of Cesàro mean and the L-curve for the deconvolution problem

In this paper, the Cesàro mean technique is applied to regularize the divergent problem which occurs in ground motion deconvolution analysis in geotechnical engineering. To deal with this ill-posed problem, we use the corner of the L-curve as the compromise point to determine the optimal order of Cesàro mean so that the high frequency content can be suppressed instead of engineering judgement using the concept of a cutoff frequency. The fractional order of Cesàro mean has been derived and used to fulfill this purpose. From the examples shown, it is found that the wave form including maximum acceleration can be accurately predicted and that both the high frequency content and divergent results can be avoided by using the proposed regularization technique.