Optimal Regularization Parameter Research Articles

Portfolio Optimization Using a Consistent Vector-Based MSE Estimation Approach

This paper is concerned with optimizing the weights of the global minimum-variance portfolio (GMVP) in high-dimensional settings where both observation and population dimensions grow at a bounded ratio. Optimizing the GMVP weights is highly influenced by the data covariance matrix estimation. In a high-dimensional setting, it is well known that the sample covariance matrix is not a proper estimator of the true covariance matrix since it is not invertible when we have fewer observations than the data dimension. Even with more observations, the sample covariance matrix may not be well-conditioned. This paper determines the GMVP weights based on a regularized covariance matrix estimator to overcome the abovementioned difficulties. Unlike other methods, the proper selection of the regularization parameter is achieved by minimizing the mean squared error of an estimate of the noise vector that accounts for the uncertainty in the data mean estimation. Using random-matrix-theory tools, we derive a consistent estimator of the achievable mean squared error that allows us to find the optimal regularization parameter using a simple line search. Simulation results demonstrate the effectiveness of the proposed method when the data dimension is larger than, or of the same order of, the number of data samples.

Spectral variable selection based on least absolute shrinkage and selection operator with ridge-adding homotopy

The least absolute shrinkage and selection operator (LASSO) is an established sparse representation approach for variable selection, and its performance relies on finding a good value for the regularization parameter, typically through cross-validation. However, cross-validation is a computationally intensive step and requires a properly determined search range and step size. In the present study, the ridge-adding homotopy (RAH) algorithm is applied with LASSO to overcome the aforementioned shortcomings. The homotopy algorithm can fit the entire solution of the LASSO problem by tracking the Karush-Kuhn Tucker (KKT) conditions and yields a finite number of potential regularization parameters. Considering the singularities, a M×1 random ridge vector will be added to the KKT conditions, which ensures that only one element is added to or removed from the active set. Finally, we can select the optimal regularization parameter by traversing the potential parameters with modelling and evaluation metrics. The selected variables are the nonzero elements in the sparse regression coefficient vector derived by the optimal regularization parameter. The proposed method has been demonstrated on three near-infrared (NIR) datasets with regard to wavelength selection and calibration. The results suggested that the “RAH-LASSO ​+ ​PLS” outperforms “LASSO ​+ ​PLS” and “full-wavelength PLS” in most cases. Importantly, the RAH method provides a systematic, as opposed to trial-and-error, procedure to determine the regularization parameter in LASSO.

Revisiting matrix-based inversion of scanning mobility particle sizer (SMPS) and humidified tandem differential mobility analyzer (HTDMA) data

Abstract. Tikhonov regularization is a tool for reducing noise amplification during data inversion. This work introduces RegularizationTools.jl, a general-purpose software package for applying Tikhonov regularization to data. The package implements well-established numerical algorithms and is suitable for systems of up to ∼ 1000 equations. Included is an abstraction to systematically categorize specific inversion configurations and their associated hyperparameters. A generic interface translates arbitrary linear forward models defined by a computer function into the corresponding design matrix. This obviates the need to explicitly write out and discretize the Fredholm integral equation, thus facilitating fast prototyping of new regularization schemes associated with measurement techniques. Example applications include the inversion involving data from scanning mobility particle sizers (SMPSs) and humidified tandem differential mobility analyzers (HTDMAs). Inversion of SMPS size distributions reported in this work builds upon the freely available software DifferentialMobilityAnalyzers.jl. The speed of inversion is improved by a factor of ∼ 200, now requiring between 2 and 5 ms per SMPS scan when using 120 size bins. Previously reported occasional failure to converge to a valid solution is reduced by switching from the L-curve method to generalized cross-validation as the metric to search for the optimal regularization parameter. Higher-order inversions resulting in smooth, denoised reconstructions of size distributions are now included in DifferentialMobilityAnalyzers.jl. This work also demonstrates that an SMPS-style matrix-based inversion can be applied to find the growth factor frequency distribution from raw HTDMA data while also accounting for multiply charged particles. The outcome of the aerosol-related inversion methods is showcased by inverting multi-week SMPS and HTDMA datasets from ground-based observations, including SMPS data obtained at Bodega Marine Laboratory during the CalWater 2/ACAPEX campaign and co-located SMPS and HTDMA data collected at the US Department of Energy observatory located at the Southern Great Plains site in Oklahoma, USA. Results show that the proposed approaches are suitable for unsupervised, nonparametric inversion of large-scale datasets as well as inversion in real time during data acquisition on low-cost reduced-instruction-set architectures used in single-board computers. The included software implementation of Tikhonov regularization is freely available, general, and domain-independent and thus can be applied to many other inverse problems arising in atmospheric measurement techniques and beyond.

A hierarchical Bayesian approach to regularization with application to the inference of relaxation spectra

The inference of the spectral function representing the relaxation process of a material is an ill-posed problem and regularization is key to solving such problems. Significant information about molecular structure can be found from the relaxation time spectra of materials, such as polymers and soft materials. Various deterministic data-driven methods including L-curve and generalized cross-validation in Tikhonov regularization have been employed in the literature to find the optimal regularization parameter. The application of Bayesian techniques for solving such ill-posed inverse problems has gained significant interest in recent years due to the increasing availability of computational resources. In this work, we formulate the inverse problem in a hierarchical Bayesian framework and consider the degree of regularization as a stochastic quantity and the relaxation spectra as the high-dimensional model parameters. Using synthetic and real data (frequency-dependent storage and loss moduli), this novel approach is employed to characterize the linear viscoelastic response of materials and obtain sparse probabilistic solutions to regression. The computations are carried out using Metropolis–Hastings-within-Gibbs sampling. Numerical results demonstrating the performance of the hierarchical Bayesian approach and comparisons with the deterministic L-curve approach are presented.

Reconstructing the nasal septum from instrument motion during septoplasty surgery

Purpose: Surgery involves modifying anatomy to achieve a goal. Reconstructing anatomy can facilitate surgical care through surgical planning, real-time decision support, or anticipating outcomes. Tool motion is a rich source of data that can be used to quantify anatomy. Our work develops and validates a method for reconstructing the nasal septum from unstructured motion of the Cottle elevator during the elevation phase of septoplasty surgery, without need to explicitly delineate the surface of the septum. Approach: The proposed method uses iterative closest point registration to initially register a template septum to the tool motion. Subsequently, statistical shape modeling with iterative most likely oriented point registration is used to fit the reconstructed septum to Cottle tip position and orientation during flap elevation. Regularization of the shape model and transformation is incorporated. The proposed methods were validated on 10 septoplasty surgeries performed on cadavers by operators of varying experience level. Preoperative CT images of the cadaver septums were segmented as ground truth. Results: We estimated reconstruction error as the difference between the projections of the Cottle tip onto the surface of the reconstructed septum and the ground-truth septum segmented from the CT image. We found translational differences of and a rotational differences of between the reconstructed septum and the ground-truth septum [median (interquartile range)], given the optimal regularization parameters. Conclusions: Accurate reconstruction of the nasal septum can be achieved from tool tracking data during septoplasty surgery on cadavers. This enables understanding of the septal anatomy without need for traditional medical imaging. This result may be used to facilitate surgical planning, intraoperative care, or skills assessment.

Using Experimentally Calibrated Regularized Stokeslets to Assess Bacterial Flagellar Motility Near a Surface

The presence of a nearby boundary is likely to be important in the life cycle and evolution of motile flagellate bacteria. This has led many authors to employ numerical simulations to model near-surface bacterial motion and compute hydrodynamic boundary effects. A common choice has been the method of images for regularized Stokeslets (MIRS); however, the method requires discretization sizes and regularization parameters that are not specified by any theory. To determine appropriate regularization parameters for given discretization choices in MIRS, we conducted dynamically similar macroscopic experiments and fit the simulations to the data. In the experiments, we measured the torque on cylinders and helices of different wavelengths as they rotated in a viscous fluid at various distances to a boundary. We found that differences between experiments and optimized simulations were less than 5% when using surface discretizations for cylinders and centerline discretizations for helices. Having determined optimal regularization parameters, we used MIRS to simulate an idealized free-swimming bacterium constructed of a cylindrical cell body and a helical flagellum moving near a boundary. We assessed the swimming performance of many bacterial morphologies by computing swimming speed, motor rotation rate, Purcell’s propulsive efficiency, energy cost per swimming distance, and a new metabolic energy cost defined to be the energy cost per body mass per swimming distance. All five measures predicted that the optimal flagellar wavelength is eight times the helical radius independently of body size and surface proximity. Although the measures disagreed on the optimal body size, they all predicted that body size is an important factor in the energy cost of bacterial motility near and far from a surface.

Error evaluation of defect shape identification using charge simulation method for static electricity

In this study, a numerical scheme for nondestructive inspection using static electricity is proposed. The relationship between the electric charges on the defect and the electric field at the observation points is described using the Fredholm integral equation for the first kind (IFK). A response function, which uses the relative coordinates between the electric charge and observation point as input variables, is derived. A system of linear equations, whose entries of the coefficient matrix are the kernels of IFK, is derived. Subsequently, an inverse algorithm is constructed to restore the electric charge distribution from the electric-field distribution. The Tikhonov regularization method is used to obtain the regularized solution because of the ill-posedness of the coefficient matrix. The optimal regularization parameter is determined using the L-curve criterion. The accuracy of the proposed method is verified through numerical experiments. The observable electric field is determined mathematically through forward analysis. A formula that quantitatively evaluates the error is then defined. It is confirmed that the inverse analysis result provides a more accurate solution than the raw data of the electric field.

C-responses estimation of geomagnetic depth sounding using regularization method and its application in Northeast China

In this paper, we introduce a technique to obtain smoother C-responses in geomagnetic depth sounding. Instead of frequency by frequency, C-responses in entire frequency range are estimated simultaneously. In order to do so, after obtaining the spectrum of Hy and Hz, the equations relating C-response with Hy and Hz are combined for a batch frequency to form a linear equation set, from which, incorporating the requirement of smooth constraint on C-responses, we could define a regularized least-squared inversion problem. And to minimize influence of noise, data squared coherencies between Z and H signals are used to weight prediction error in order to suppress unreliable data. Synthetic data tests indicate that using the optimal regularization parameter determined by V-curve method other than L-curve method, regularization method could produce rather smooth C-responses from noise interfered signals. C-responses in Northeast China are thus obtained and then 1-D inversion is applied, from the obtained electrical and combining the Global and regional tomographic results, it is speculated that the stagnant plate in MTZ leads the high conductivity anomaly in Northeast China.

Two-dimensional NMR inversion based on fast norm smoothing method

Two-dimensional (2D) nuclear magnetic resonance (NMR) inversion operates with massive echo train data and is an ill-posed problem. It is very important to select a suitable inversion method for the 2D NMR data processing. In this study, we propose a fast, robust, and effective method for 2D NMR inversion that improves the computational efficiency of the inversion process by avoiding estimation of some unneeded regularization parameters. Firstly, a method that combines window averaging (WA) and singular value decomposition (SVD) is used to compress the echo train data and obtain the singular values of the kernel matrix. Subsequently, an optimum regularization parameter in a fast manner using the signal-to-noise ratio (SNR) of the echo train data and the maximum singular value of the kernel matrix are determined. Finally, we use the Butler-Reeds-Dawson (BRD) method and the selected optimum regularization parameter to invert the compressed data to achieve a fast 2D NMR inversion. The numerical simulation results indicate that the proposed method not only achieves satisfactory 2D NMR spectra rapidly from the echo train data of different SNRs but also is insensitive to the number of the final compressed data points.

Image quality guided iterative reconstruction for low-dose CT based on CT image statistics

Iterative reconstruction frameworks show predominance in low dose and incomplete data situations. In the iterative reconstruction framework, there are two components, i.e., the fidelity term aims to maintain the structure details of the reconstructed object, and the regularization term uses prior information to suppress the artifacts such as noise. A regularization parameter balances them, aiming to find a good trade-off between noise and resolution. Currently, the regularization parameters are selected as a rule of thumb or some prior knowledge assumption is required, which limits practical uses. Furthermore, the computation cost of regularization parameter selection is also heavy. In this paper, we address this problem by introducing CT image quality assessment (IQA) into the iterative reconstruction framework. Several steps are involved during the study. First, we analyze the CT image statistics using the dual dictionary (DDL) method. Regularities are observed and concluded, revealing the relationship among the regularization parameter, iterations, and CT image quality. Second, with derivation and simplification of DDL procedure, a CT IQA metric named SODVAC is designed. The SODVAC locates the optimal regularization parameter that results in the reconstructed image with distinct structures and with no noise or little noise. Thirdly, we introduce SODVAC into the iterative reconstruction framework and then propose a general image-quality-guided iterative reconstruction (QIR) framework and give a specific framework example (sQIR) by introducing SODVAC into the iterative reconstruction framework. sQIR simultaneously optimizes the reconstructed image and the regularization parameter during the iterations. Results confirm the effectiveness of the proposed method. No prior information is needed and the low computation cost are the advantages of our method compared with existing state-of-the-art L-curve and ZIP selection strategies.

Time-Domain Multidimensional Deconvolution: A Physically Reliable and Stable Preconditioned Implementation

Multidimensional deconvolution constitutes an essential operation in a variety of geophysical scenarios at different scales ranging from reservoir to crustal, as it appears in applications such as surface multiple elimination, target-oriented redatuming, and interferometric body-wave retrieval just to name a few. Depending on the use case, active, microseismic, or teleseismic signals are used to reconstruct the broadband response that would have been recorded between two observation points as if one were a virtual source. Reconstructing such a response relies on the the solution of an ill-conditioned linear inverse problem sensitive to noise and artifacts due to incomplete acquisition, limited sources, and band-limited data. Typically, this inversion is performed in the Fourier domain where the inverse problem is solved per frequency via direct or iterative solvers. While this inversion is in theory meant to remove spurious events from cross-correlation gathers and to correct amplitudes, difficulties arise in the estimation of optimal regularization parameters, which are worsened by the fact they must be estimated at each frequency independently. Here we show the benefits of formulating the problem in the time domain and introduce a number of physical constraints that naturally drive the inversion towards a reduced set of stable, meaningful solutions. By exploiting reciprocity, time causality, and frequency-wavenumber locality a set of preconditioners are included at minimal additional cost as a way to alleviate the dependency on an optimal damping parameter to stabilize the inversion. With an interferometric redatuming example, we demonstrate how our time domain implementation successfully reconstructs the overburden-free reflection response beneath a complex salt body from noise-contaminated up- and down-going transmission responses at the target level.

A New Hybrid Method for Solving Inverse Heat Conduction Problems

Solving the inverse problems, especially in the field of heat transfer, is one of the challenges of engineering due to its importance in industrial applications. It is well-known that inverse heat conduction problems (IHCPs) are severely ill-posed, which means that small disturbances in the input may cause extremely large errors in the solution. This paper introduces an accurate method for solving inverse problems by combining Tikhonov's regularization and the genetic algorithm. Finding the regularization parameter as the decisive parameter is modelled by this method, a few sample problems were solved to investigate the efficiency and accuracy of the proposed method. A linear sum of fundamental solutions with unknown constant coefficients assumed as an approximated solution to the sample IHCP problem and collocation method is used to minimize residues in the collocation points. In this contribution, we use Morozov's discrepancy principle and Quasi-Optimality criterion for defining the objective function, which must be minimized to yield the value of the optimum regularization parameter.

Optimal Regularization Method Based on the L-Curve for Solving Rational Function Model Parameters

Rational polynomial coefficients in a rational function model (<small>RFM</small>) have high correlation and redundancy, especially in high-order <small>RFMs</small>, which results in ill-posed problems of the normal equation. For this reason, this article presents an optimal regularization method with the L-curve for solving rational polynomial coefficients. This method estimates the rational polynomial coefficients of an <small>RFM</small> using the L-curve and finds the optimal regularization parameter with the minimum mean square error, then solves the parameters of the <small>RFM</small> by the Tikhonov method based on the optimal regularization parameter. The proposed method is validated in both terrain-dependent and terrain-independent cases using Gaofen-1 and aerial images, respectively, and compared with the least-squares method, L-curve method, and generalized cross-validation method. The experimental results demonstrate that the proposed method can solve the <small> RFM</small> parameters effectively, and their accuracy is increased by more than 85% on average relative to the other methods.

Terahertz nondestructive quantitative characterization for layer thickness based on sparse representation method

Terahertz (THz) time-domain detection has emerged as a promising tool for nondestructive testing due to its superior detection accuracy and sensitivity for both interfaces and internal defects inside non-metallic materials. In THz nondestructive evaluation, accurate characterizations of time-of-flight (TOF) extracted from the echo signal are critical to locate the layer interfaces and damages inside the structure. However, for prevalent complex THz echo signals such as overlapping echoes, low signal-to-noise (SNR) echoes and severe dispersion echoes, conventional methods by merely determining the TOF such as the direct method and the deconvolution method might be compromised. Here we have proposed the sparse representation (SR) method to quantitatively characterize the layer thickness by reconstructing the impulse response from complex THz echo signals. To balance the accuracy and the computational efficiency, we have proposed a combinational search method for selecting the optimal regularization parameter and constructed a convolution dictionary based on the structural prior of THz reference echo during the sparse decomposition process. Both numerical and experimental analysis are performed to validate the robustness of SR method on the THz echo localization compared with those conventional methods. It can be expected that SR will provide a novel approach to the application of precise localization and quantification of defects in THz based nondestructive techniques.

Impact Force Reconstruction of Composite materials based on Improved Regularization Technology

In the structural health monitoring of composite materials, in order to solve the ill-posed problem of impact force reconstruction, regularization techniques are often used to deal with it. Due to the poor convergence of the traditional Tikhonov regularization method, in order to accurately reconstruct the time history of the impact force, this paper improves Tikhonov regularization method and constructs homotopy function with strong convergence. Since the optimal regularization parameters need to be found in the homotopy function, the Newton downhill method is used to find the optimal parameters and the homotopy function can be calculated, which can accurately reconstruct the time history of the impact force. In order to verify the universality of the method in this paper, impact hammers of different materials were used in the experiment in this paper to study and compare the reconstruction effect of impact time history of different impact hammers.

Combining instrument inversions for sub-10 nm aerosol number size-distribution measurements

Resolving aerosol dynamical processes in the sub-10 nm range is crucial for our understanding of the contribution of new particle formation to the global cloud condensation nuclei budget or air pollution. Accurate measurements of the particle size distribution in this size-range are challenging due to high diffusional losses and low charging and/or detection efficiencies. Several instruments have been developed in recent years in order to access the sub-10 nm particle size distribution; however, no single instrument can provide high counting statistics, low systematic uncertainties and high size-resolution at the same time. Here we compare several data inversion approaches that allow combining data from different sizing instruments during the inversion and provide python/Julia packages for free usage of the methods. We find that Tikhonov regularization using the L-curve method for optimal regularization parameter estimation gives very reliable results over a wide range of tested data sets and clearly improves standard inversion approaches. Kalman Filtering or regularization using a Poisson likelihood can be powerful tools, especially for well-defined chamber experiments or data from mobility spectrometers only, respectively. Nullspace optimization and non-linear iterative regression are clearly inferior compared to the aforementioned methods. We show that with regularization we can reconstruct the size-distribution measured by up to 4 different mobility particle size spectrometer systems and several particle counters for datasets from Hyytiälä and Helsinki, Finland, revealing the sub-10 nm aerosol dynamics in more detail compared to a single instrument assessment.

An intelligent method based on feed-forward artificial neural network and least square support vector machine for the simultaneous spectrophotometric estimation of anti hepatitis C virus drugs in pharmaceutical formulation and biological fluid

This study proposed simple and reliable spectrophotometry method for simultaneous analysis of hepatitis C antiviral binary mixture containing sofosbuvir (SOF) and daclatasvir (DAC). This technique is based on the use of feed-forward artificial neural network (FF-ANN) and least square support vector machine (LS-SVM). FF-NN with Levenberg–Marquardt (LM) and Cartesian genetic programming (CGP) algorithms was trained to determine the best number of hidden layers and the number of neurons. This comparison demonstrated that the LM algorithm had the minimum mean square error (MSE) for SOF (1.59 × 10−28) and DAC (4.71 × 10−28). In LS-SVM model, the optimum regularization parameter (γ) and width of the function (σ) were achieved with root mean square error (RMSE) of 0.9355 and 0.2641 for SOF and DAC, respectively. The coefficient of determination (R2) value of mixtures containing SOF and DAC was 0.996 and 0.997, respectively. The percentage recovery values were in the range of 94.03–104.58 and 94.04–106.41 for SOF and DAC, respectively. Statistical test (ANOVA) was implemented to compare high-performance liquid chromatography (HPLC) and spectrophotometry, which showed no significant difference. These results indicate that the proposed method possesses great potential ability for prediction of concentration of components in pharmaceutical formulations.

Double Gaussian mixture model-based terahertz wave dispersion compensation method using convex optimization technique

Terahertz (THz) time-domain pulse detection technique has recently attracted great attention in the nondestructive testing (NDT) field due to its superior detection accuracy and sensitivity for various non-metallic materials. However, the dispersion of the THz wave in the tested material significantly degrades the temporal and spatial resolution of THz detection, which limits its practical applications in the high-resolution measurement. Here, a novel dispersion compensation strategy based on the double Gaussian mixture model (DGMM) is proposed to suppress the dispersion effect in THz echoes. The double overcomplete dictionaries composed of one parametric dispersion dictionary and one parametric non-dispersion dictionary are designed separately by discretizing the estimated parameter vector from observed THz echoes. To obtain the excellent performance of dispersion compensation, the convex optimization technique is employed to sparsely decompose the observed dispersive THz echoes by the appropriate selection of the optimal regularization parameter. Both numerical simulations and experiments can validate the effectiveness of proposed method for the non-overlapping and overlapping dispersive THz echoes.

Estimating the forcing function in a mechanical system by an inverse calibration method

This article proposes and demonstrates a calibration-based integral formulation for resolving the forcing function in a mass–spring–damper system, given either displacement or acceleration data. The proposed method is novel in the context of vibrations, being thoroughly studied in the field of heat transfer. The approach can be expanded and generalized further to multi-variable systems associated with machine parts, vehicle suspensions, translational and rotational systems, gear systems, etc. when mathematically described by a system of constant property, linear, time-invariant ordinary differential equations. The analytic approach and subsequent numerical reconstruction of the forcing function is based on resolving a parameter-free inverse formulation for the equation(s) of motion. The calibration approach is formulated in the frequency domain and takes advantage of several observations produced by the dimensionality reduction leading to an algebratized system involving an input–output relationship and a transfer function possessing all the system parameters. The transfer function is eliminated in lieu of experimental data, from a calibration effort, thus leading to a reduction of systematic errors. These parameter-free, reduced systematic error aspects are the distinct and novel advantages of the proposed method. A first-kind Volterra integral equation is formed containing only the unknown forcing function and experimental data. As with all ill-posed problems, regularization must be introduced for system stabilization. A future-time technique is instituted for forming a family of predictions based on the chosen regularization parameter. The optimal regularization parameter is estimated using a combination of phase–plane analysis and cross-correlation principles. Finally, a numerical simulation is performed verifying the proposed approach.

Improved Generalized Cross-Validation and Unbiased Predictive Risk Estimator Methods Using the RGSVD: Application to Inversion of Potential Field Data

The inversion of potential field data has widely utilized the generalized cross-validation (GCV) and the unbiased predictive risk estimator (UPRE) methods to determine the regularization parameter. However, these two methods are time-consuming and it is difficult for them to determine the optimal linear search range including the optimal regularization. To solve these problems, this article improves the GCV and UPRE methods using the RGSVD (randomized generalized singular value decomposition) algorithm. The improved methods first use the randomized algorithm to compute an approximate generalized singular value decomposition (GSVD) with less computational time. Then, the optimal linear search range is determined based on the generalized singular values. Finally, the GCV and the UPRE functions are efficiently computed on the basis of the results from the RGSVD algorithm. In this way, the GCV and UPRE methods using the RGSVD algorithm are able to determine the optimal regularization parameter fast and effectively. One comparative test shows the effectiveness and efficiency of the GCV and the UPRE methods using the RGSVD algorithm.