Regularized Cost Function Research Articles

A Nonparametric Regularization for Spectrum Estimation of Time-Varying Output-Only Measurements

In this work, an advanced 2D nonparametric correlogram method is presented to cope with output-only measurements of linear (slow) time-varying systems. The proposed method is a novel generalization of the kernel function-based regularization techniques that have been developed for estimating linear time-invariant impulse response functions. In the proposed system identification technique, an estimation method is provided that can estimate the time-varying auto- and cross-correlation function and indirectly, the time-varying auto- and cross-correlation power spectrum estimates based on real-life measurements without measuring the perturbation signals. The (slow) time-varying behavior means that the dynamic of the system changes as a function of time. In this work, a tailored regularization cost function is considered to impose assumptions such as smoothness and stability on the 2D auto- and cross-correlation function resulting in robust and uniquely determined estimates. The proposed method is validated on two examples: a simulation to check the numerical correctness of the method, and a flutter test measurement of a scaled airplane model to illustrate the power of the method on a real-life challenging problem.

Regularized cost function in wavefront shaping for advancing the contrast of structured light.

The cost function in the iterative optimization algorithms is one of the sensitive optimization controllers that plays a crucial role in feedback based wavefront shaping for constructing well-resolved complex structured light through scattering media. There has been a trade-off between resolution and the contrast enhancement of the structured light in wavefront shaping. We have developed an ℓ 2-norm based quadratic cost function (L2QN) and proposed a regularized cost function (RCF) for advancing the contrast and maintaining the high resolution of structured light. Both the simulations and experiments have been performed, and it has been found that the proposed RCF significantly advances the contrast and structural uniformity for focusing light through scattering media as well as for diffused reflection mode. The potential applications of the method demonstrated in this study can be extended into holographic displays, structured light illumination microscopy, photo-lithography, photothermal treatments, dosimetry, laser materials processing, and energy control inside and outside an incubation system.

Particle-Based Optimization Algorithms for Longitudinal Control of Autonomous Vehicle: A Comparative Study

In order to improve the stability and performance of an autonomous vehicle, optimization needs to be explicitly performed in the controllers, which has an essential part in the tracking system. This work proposes a novel longitudinal control optimization scheme and a novel longitudinal controller consisting of a feed-forward and feedback term. The feed-forward term is inspired by the vehicle’s steady-state response, whereas the feedback term is a proportional-integral-derivative (PID) controller. Also, a model representing the longitudinal vehicle dynamics is designed based on physical phenomena affecting the vehicle. Besides, some nature-inspired optimization algorithms are used to determine the optimal model parameters and optimize the controller parameters, i.e., Particle Swarm Optimization (PSO), Accelerated PSO (APSO), Flower Pollination Algorithm (FPA), and Modified FPA (MFPA). The algorithms are compared in optimizing the longitudinal vehicle model and controller using the CARLA simulator, and stability tests are also done for each algorithm. In addition, the characteristics of several cost functions in controller optimization are inspected. The results show that the MFPA is the most stable algorithm, the proposed model represents the system satisfactorily, and optimizing the controller using a regularized cost function leads to better overall performance. Our code is available in https://github.com/fadamsyah/Particle-Based-Optimization-for-Longitudinal-Control.

On an inverse problem for a nonlinear third order in time partial differential equation

In this article, first we convert an inverse problem of determining the unknown timewise terms of nonlinear third order in time partial differential equation (PDE) from knowledge of two boundary measurements to the auxiliary system of integral equations. Then, existence and uniqueness of the solution of this system is proved by means of the contraction principle on a small time interval. Also uniqueness of the solution of the inverse problem is given. However, since the governing equation is yet ill-posed (very slight errors in the temperature input may cause relatively significant errors in the output potential and source terms), we need to regularize the solution. Therefore, to get a stable solution, a regularized cost function is to be minimized for retrieval of the unknown terms. The third order in time PDE problem is discretized using the FDM and reshaped as non-linear least-squares optimization of the Tikhonov regularization function. This is numerically solved by means of the MATLAB subroutine lsqnonlin tool. Both analytical and perturbed data are inverted. Numerical outcomes for two benchmark test examples are reported and discussed. The proposed numerical approach has also been discussed.

Censoring outliers in elliptically symmetric multivariate distribution

In this paper, we discuss the problem of censoring outliers in a dataset with a class of complex elliptically symmetric (CES) distribution, which has various engineering applications. To this end, the regularized cost function is formulated by the maximum likelihood (ML) setting. Since the ML estimation of the free outlier subset requires solving a combinatorial problem, an efficient and low complexity algorithm is proposed that guarantees the convergence to a stationary point. Two independent and dependent penalty terms to the selected subset are considered. The conventional generalized inner product (GIP) and the submodular minimization approaches are employed in the derivation of the proposed algorithm in the independent and dependent cases, respectively. The dependent case improves performance by slightly increasing complexity. As a special case for Gaussian assumption and using a non-regularized cost function, the proposed algorithm reduces to the fast minimum covariance determinant (FMCD) algorithm, which has impressive efficiency and received considerable attention in statistics and computer science. Hence, the proposed algorithm extends/generalizes the FMCD to be applicable to the non-Gaussian distributions. The simulation results highlight that the proposed algorithms exhibit satisfactory performance in CES environments.

Incorporating Gradient Similarity for Robust Time Delay Estimation in Ultrasound Elastography.

Energy-based ultrasound elastography techniques minimize a regularized cost function consisting of data and continuity terms to obtain local displacement estimates based on the local time-delay estimation (TDE) between radio frequency (RF) frames. The data term associated with the existing techniques takes only the amplitude similarity into account and hence is not sufficiently robust to the outlier samples present in the RF frames under consideration. This drawback creates noticeable artifacts in the strain image. To resolve this issue, we propose to formulate the data function as a linear combination of the amplitude and gradient similarity constraints. We estimate the adaptive weight concerning each similarity term following an iterative scheme. Finally, we optimize the nonlinear cost function in an efficient manner to convert the problem to a sparse system of linear equations which are solved for millions of variables. We call our technique rGLUE: robust data term in GLobal Ultrasound Elastography. rGLUE has been validated using simulation, phantom, in vivo liver, and breast datasets. In all our experiments, rGLUE substantially outperforms the recent elastography methods both visually and quantitatively. For simulated, phantom, and in vivo datasets, respectively, rGLUE achieves 107%, 18%, and 23% improvements of signal-to-noise ratio (SNR) and 61%, 19%, and 25% improvements of contrast-to-noise ratio (CNR) over global ultrasound elastography (GLUE), a recently published elastography algorithm.

Recovering a source term in the higher-order pseudo-parabolic equation via cubic spline functions

In this paper, we considered an inverse problem of recovering the space-dependent source coefficient in the third-order pseudo-parabolic equation from final over-determination condition. This inverse problem appears extensively in the modelling of various phenomena in physics such as the motion of non-Newtonian fluids, thermodynamic processes, filtration in a porous medium, etc. The unique solvability theorem for this inverse problem is supplied. However, since the governing equation is yet ill-posed (very slight errors in the final input may cause relatively significant errors in the output source term), we need to regularize the solution. Therefore, to get a stable solution, a regularized cost function is to be minimized for retrieval of the unknown force term. The third-order pseudo-parabolic problem is discretized using the Cubic B-spline (CB-spline) collocation technique and reshaped as non-linear least-squares optimization of the Tikhonov regularization function. Numerically, this is effectively solved using the lsqnonlin routine from the MATLAB toolbox. Both perturbed data and analytical solutions are inverted. Numerical outcomes are reported and discussed. The computational efficiency of the method is investigated by small values of CPU time. In addition, the von Neumann stability analysis for the proposed numerical approach has also been discussed.

Algorithmic-gradient approach for the Price of Anarchy and Stability for incomplete information

In game theory, the Price of Anarchy (PoA) and the Price of Stability (PoS) are excellent tools for characterizing the loss of competition’s efficiency. This paper investigates the PoA and PoS for incomplete information in Bayesian-Markov games. The development of computationally effective optimization tools is essential for the investigation of these equilibria. To attain this goal, the difficulty of computing the PoA and PoS is presented as a tractable nonlinear programming problem. We look at regularized cost-minimization issues and present a framework for building projection-gradient algorithms that have near-optimal a priori performance guarantees. For the constrained Lagrange technique, we use Tikhonov’s regularization approach, for which the original problem converges to a single solution. Then we suggest a framework for computing the difference between the ‘best equilibrium’ and the ‘optimal solution’, as well as the ratio between the ‘worst equilibrium’ and the ‘optimal solution’. To solve these issues, we use a game-theoretic method, assigning agents cost functions to maximize (PoA) or minimize (PoS) the equilibrium efficiency. After the agents’ regularized-cost functions have been established, every algorithm capable of discovering a Nash equilibrium of the system inherits a performance guarantee matching the PoA and Pos. The convergence of the suggested projection-gradient method under mild conditions is established. Finally, we consider the implications of our findings when this paradigm is applied to convex systems.

Controlling a random population

Bertrand et al. introduced a model of parameterised systems, where each agent is represented by a finite state system, and studied the following control problem: for any number of agents, does there exist a controller able to bring all agents to a target state? They showed that the problem is decidable and EXPTIME-complete in the adversarial setting, and posed as an open problem the stochastic setting, where the agent is represented by a Markov decision process. In this paper, we show that the stochastic control problem is decidable. Our solution makes significant uses of well quasi orders, of the max-flow min-cut theorem, and of the theory of regular cost functions. We introduce an intermediate problem of independence interest called the sequential flow problem and study its complexity.

Proximal Algorithms for Discrete-Level Phase-Shifting Mask Design with Application to Optogenetics

This work studies the problem of designing computer-generated holograms using phase-shifting masks limited to represent only a small number of discrete phase levels. This problem has various applications, notably in the emerging field of optogenetics and lithography. A novel regularized cost function is proposed for the problem at hand that penalizes the unfeasible phase levels. Since the proposed cost function is non-smooth, we derive proper proximal gradient algorithms for its minimization. Simulation results, considering an optogenetics application, demonstrate that the proposed proximal gradient algorithm yields better performance as compared to other algorithms proposed in the literature.

Analytic Global Regularized Backscatter Quantitative Ultrasound.

Although a variety of techniques have been developed to reduce the appearance of B-mode speckle, quantitative ultrasound (QUS) aims at extracting the hidden properties of the tissue. Herein, we propose two novel techniques to accurately and precisely estimate two important QUS parameters, namely, the average attenuation coefficient and the backscatter coefficient. Both the techniques optimize a cost function that incorporates data and continuity constraint terms, which we call AnaLytical Global rEgularized BackscatteR quAntitative ultrasound (ALGEBRA). We propose two versions of ALGEBRA, namely, 1-D- and 2-D-ALGEBRA. In 1-D-ALGEBRA, the regularized cost function is formulated in the axial direction, and the QUS parameters are calculated for one line of radio frequency (RF) echo data. In 2-D-ALGEBRA, the regularized cost function is formulated for the entire image, and the QUS parameters throughout the image are estimated simultaneously. This simultaneous optimization allows 2-D-ALGEBRA to "see" all the data before estimating the QUS parameters. In both the methods, we efficiently optimize the cost functions by casting it as a sparse linear system of equations. As a result of this efficient optimization, 1-D-ALGEBRA and 2-D-ALGEBRA are, respectively, 600 and 300 times faster than optimization using the dynamic programming (DP) method previously proposed by our group. In addition, the proposed technique has fewer input parameters that require manual tuning. Our results demonstrate that the proposed ALGEBRA methods substantially outperform least-square and DP methods in estimating the QUS parameters in phantom experiments.

Novel iterative ensemble smoothers derived from a class of generalized cost functions

Iterative ensemble smoothers (IES) are among the state-of-the-art approaches to solving history matching problems. From an optimization-theoretic point of view, these algorithms can be derived by solving certain stochastic nonlinear-least-squares problems. In a broader picture, history matching is essentially an inverse problem, which is often ill-posed and may not possess a unique solution. To mitigate the ill-posedness, in the course of solving an inverse problem, prior knowledge and domain experience are often incorporated, as a regularization term, into a suitable cost function within a respective optimization problem. Whereas in the inverse theory there is a rich class of inversion algorithms resulting from various choices of regularized cost functions, there are few ensemble data assimilation algorithms (including IES) which in their practical uses are implemented in a form beyond nonlinear-least-squares. This work aims to narrow this noticed gap. Specifically, we consider a class of more generalized cost functions, and establish a unified formula that can be used to construct a corresponding group of novel ensemble data assimilation algorithms, called generalized IES (GIES), in a principled and systematic way. For demonstration, we choose a subset (up to 30 +) of the GIES algorithms derived from the unified formula, and apply them to two history matching problems. Experiment results indicate that many of the tested GIES algorithms exhibit superior performance to that of an original IES developed in a previous work, showcasing the potential benefit of designing new ensemble data assimilation algorithms through the proposed framework.

Two-level monotonic multistage recommender systems

A recommender system learns to predict the user-specific preference or intention over many items simultaneously for all users, making personalized recommendations based on a relatively small number of observations. One central issue is how to leverage three-way interactions, referred to as user-item-stage dependencies on a monotonic chain of events, to enhance the prediction accuracy. A monotonic chain of events occurs, for instance, in an article sharing dataset, where a “follow” action implies a “like” action, which in turn implies a “view” action. In this article, we develop a multistage recommender system utilizing a two-level monotonic property characterizing a monotonic chain of events for personalized prediction. Particularly, we derive a large-margin classifier based on a nonnegative additive latent factor model in the presence of a high percentage of missing observations, particularly between stages, reducing the number of model parameters for personalized prediction while guaranteeing prediction consistency. On this ground, we derive a regularized cost function to learn user-specific behaviors at different stages, linking decision functions to numerical and categorical covariates to model user-item-stage interactions. Computationally, we derive an algorithm based on blockwise coordinate descent. Theoretically, we show that the two-level monotonic property enhances the accuracy of learning as compared to a standard method treating each stage individually and an ordinal method utilizing only one-level monotonicity. Finally, the proposed method compares favorably with existing methods in simulations and an article sharing dataset.

Enhancing Function Approximation Abilities of Neural Networks by Training Derivatives.

A method to increase the precision of feedforward networks is proposed. It requires prior knowledge of a target's function derivatives of several orders and uses this information in gradient-based training. Forward pass calculates not only the values of the output layer of a network but also their derivatives. The deviations of those derivatives from the target ones are used in an extended cost function, and then, the backward pass calculates the gradient of the extended cost with respect to weights, which is then used by a weights update algorithm. The most accurate approximation is obtained when the training starts with all available derivatives that are then step by step excluded from the extended cost function, starting with the highest orders up until only values are trained. Despite a substantial increase in arithmetic operations per pattern (compared with the conventional training), the method allows to obtain 140-1000 times more accurate approximation for simple cases if the total number of operations is equal. This precision also happens to be out of reach for the regular cost function. The method works well for solving differential equations with neural networks. The cost function is the deviation of the equation's residual from zero, and it can be extended by differentiating the equation itself so no prior information is required. This extension allows to solve 2-D nonlinear partial differential equation 13 times more accurately using seven times fewer grid points. The GPU-efficient algorithm for calculating the gradient of the extended cost function is proposed.

AIRRLS: An Augmented Iteratively Re-weighted and Refined Least Squares Algorithm for Inverse Modeling of Magnetometry Data

This work aims to examine the functionality of a new Augmented Iteratively Re-weighted and Refined Least Squares algorithm (AIRRLS) togenerate a 3D model of magnetic susceptibility property from a potentialfield magnetometry survey. Whereby this algorithm ameliorates an lpnorm Tikhonov regularization cost function through replacing a set ofweighted linear system of equations. It leads to constructing a magneticsusceptibility model that iteratively converges to an optimum solution,meanwhile the regularization parameter performs as a stopping criterionto finalize the iterations. To tackle and suppress the intrinsic tendency ofa sought target responsible for generating a magnetic anomaly and to notbe imaged at shallow depth in inverse modeling, a prior depth weightingfunction is imposed in the principle system of equations. The significanceof this research lies in improvement of the performance of the inversion,where the running time of an lp norm problem after incorporating apre-conditioner conjugate gradient solver (PCCG) in cases of large scalegeophysical dataset. Forasmuch as this study attempts to image a geological target with low magnetic susceptibility property, it is assumed thatthere is no remanent magnetization. The applicability of the algorithm istested for a synthetic multi-source data to demonstrate its performancein 3D modeling . Subsequently, a real case study in Semnan provinceof Iran, is investigated to image an embedded porphyry copper layerin a sequence of sediments. The sought target consists of a concealedarc-shaped porphyry andesite unit that may have potential of Cuoccurrences. Results prove that it extends down at depth, so exploratorydrilling is highly recommended to get insights about its potential forCu-bearing mineralization.

Non-blind image blur removal method based on a Bayesian hierarchical model with hyperparameter priors

In many image blur removal schemes, a proper point spread function is usually estimated in advance from the blurry image, then the latent image comes out by using existing non-blind techniques. However, some of the techniques suffer from strong artifacts. Therefore, an efficient non-blind method plays an important role in image restoration issues. In most models, image priors act as the regularization terms that hold image details and suppress noises. This paper introduces a new image prior based on a parameterized scaled Gaussian model and a gamma distribution, with hyperparameters based on the statistical properties of tens of thousands of images. Our regularized cost function is then formed via a Bayesian hierarchical approach. It consists of a data fidelity term and a series of constraints on image gradients in multiple orientations. The former is used to assure the best approximation of the original image, and the latter is for preserving sharp edges. The optimization problem is solved by an effective tail-recursive algorithm based on the conjugate descent technique. Experimental results show that our model can both deal with simulated data and real scenes. The comparisons show our method outperforms others and achieves promising results.

Hierarchical Trellis Based Decoder for Total Variation Sequence Detection (TVSD) in Space-Time Trellis Coded (STTC) Wireless Image/Video Communication

This paper proposes a robust reconstruction scheme for image/ video transmission in space-time trellis coding (STTC) based MIMO wireless multimedia sensor networks (WMSN). The information bits of the image/ video stream are modulated using STTC prior to transmission over the MIMO wireless channel between the multimedia sensors and the cluster head. At the receiver, a novel total variation sequence detector (TVSD) is developed that employs an anisotropic total variation (TV) regularized cost function to leverage the bounded variation property of the image/ video stream, in addition to the coding and diversity advantages of the STTC, for improved image/ video recovery. For reconstruction, a novel bi-layered hierarchical trellis based modified Viterbi decoder is developed to decode the optimal bit sequence corresponding to the least TV cost function using appropriately modified node and branch metrics. Simulation results using several test images and video sequences demonstrate the improved reconstruction performance of the proposed scheme, especially in the low SNR regime (0 - 10 dB), in comparison to the conventional maximum likelihood (ML) decoder, which makes it ideally suited for implementation in practical WMSN.

Sound Deposit Insurance Pricing Using a Machine Learning Approach

While the main conceptual issue related to deposit insurances is the moral hazard risk, the main technical issue is inaccurate calibration of the implied volatility. This issue can raise the risk of generating an arbitrage. In this paper, first, we discuss that by imposing the no-moral-hazard risk, the removal of arbitrage is equivalent to removing the static arbitrage. Then, we propose a simple quadratic model to parameterize implied volatility and remove the static arbitrage. The process of removing the static risk is as follows: Using a machine learning approach with a regularized cost function, we update the parameters in such a way that butterfly arbitrage is ruled out and also implementing a calibration method, we make some conditions on the parameters of each time slice to rule out calendar spread arbitrage. Therefore, eliminating the effects of both butterfly and calendar spread arbitrage make the implied volatility surface free of static arbitrage.

Sparsity-enhanced signal decomposition via generalized minimax-concave penalty for gearbox fault diagnosis

Vibration signals arising from faulty gearboxes are often a mixture of the meshing component and the periodic transient component, and simultaneously contaminated by noise. Sparsity-assisted signal decomposition is an effective technique to decompose a signal into morphologically distinct components based on sparse representation and optimization. In this paper, we propose a sparsity-enhanced signal decomposition method which uses the generalized minimax-concave (GMC) penalty as a nonconvex regularizer to enhance sparsity in the sparse approximation compared to classical sparsity-assisted signal decomposition methods, and thus to improve the decomposition accuracy for gearbox fault diagnosis. Even though the GMC penalty itself is nonconvex, it maintains the convexity of the GMC regularized cost function to be minimized. Hence, similar to the classical L1-norm regularization methods, the global optimal solution can be guaranteed via convex optimization. Moreover, we present and validate a straight-forward way to choose transforms and set parameters for the proposed method. Through simulation studies, it is demonstrated that the proposed sparsity-enhanced signal decomposition method can effectively decompose the simulated faulty gearbox signal into the meshing component and the periodic transient component. Comparisons with the classical L1-norm regularized signal decomposition method and spectral kurtosis show that the proposed method can accurately preserve the amplitude of the periodic transient component and provide a more accurate estimation result. Experiment and engineering case studies further verify that the proposed method can accurately estimate the periodic transient component from vibration signals, which demonstrate that the proposed method is a promising tool for gearbox fault diagnosis.

Optimality and identification of dynamic models in systems biology: an inverse optimal control framework.

Optimality principles have been used to explain many biological processes and systems. However, the functions being optimized are in general unknown a priori. Here we present an inverse optimal control framework for modeling dynamics in systems biology. The objective is to identify the underlying optimality principle from observed time-series data and simultaneously estimate unmeasured time-dependent inputs and time-invariant model parameters. As a special case, we also consider the problem of optimal simultaneous estimation of inputs and parameters from noisy data. After presenting a general statement of the inverse optimal control problem, and discussing special cases of interest, we outline numerical strategies which are scalable and robust. We discuss the existence, relevance and implications of identifiability issues in the above problems. We present a robust computational approach based on regularized cost functions and the use of suitable direct numerical methods based on the control-vector parameterization approach. To avoid convergence to local solutions, we make use of hybrid global-local methods. We illustrate the performance and capabilities of this approach with several challenging case studies, including simulated and real data. We pay particular attention to the computational scalability of our approach (with the objective of considering large numbers of inputs and states). We provide a software implementation of both the methods and the case studies. The code used to obtain the results reported here is available at https://zenodo.org/record/1009541. Supplementary data are available at Bioinformatics online.