Ill-posed Optimization Problem Research Articles

Active control of electromagnetic fields in layered media

This article presents a numerical strategy for actively manipulating electromagnetic (EM) fields in layered media. In particular, we develop a scheme to characterize an EM source that will generate some predetermined field patterns in prescribed disjoint exterior regions in layered media. The proposed question of specifying such an EM source is not an inverse source problem (ISP) since the existence of a solution is not guaranteed. Moreover, our problem allows for the possibility of prescribing different EM fields in mutually disjoint exterior regions. This question involves a linear inverse problem that requires solving a severely ill-posed optimization problem (i.e. suffering from possible non-existence or non-uniqueness of a solution). The forward operator is defined by expressing the EM fields as a function of the current at the source using the layered media Green's function (LMGF), accounting for the physical parameters of the layered media. This results to integral equations that are then discretized using the method of moments (MoM), yielding an ill-posed system of linear equations. Unlike in ISPs, stability with respect to data is not an issue here since no data is measured. Rather, stability with respect to input current approximation is important. To get such stable solutions, we applied two regularization methods, namely, the truncated singular value decomposition (TSVD) method and the Tikhonov regularization method with the Morozov Discrepancy Principle. We performed several numerical simulations to support the theoretical framework and analyzes, and to demonstrate the accuracy and feasibility of the proposed numerical algorithms.

Robust Clustering with Normal Mixture Models: A Pseudo [formula omitted]-Likelihood Approach

As in other estimation scenarios, likelihood based estimation in the normal mixture set-up is highly non-robust against model misspecification and presence of outliers (apart from being an ill-posed optimization problem). A robust alternative to the ordinary likelihood approach for this estimation problem is proposed which performs simultaneous estimation and data clustering and leads to subsequent anomaly detection. To invoke robustness, the methodology based on the minimization of the density power divergence (or alternatively, the maximization of the β-likelihood) is utilized under suitable constraints. An iteratively reweighted least squares approach has been followed in order to compute the proposed estimators for the component means (or equivalently cluster centers) and component dispersion matrices which leads to simultaneous data clustering. Some exploratory techniques are also suggested for anomaly detection, a problem of great importance in the domain of statistics and machine learning. The proposed method is validated with simulation studies under different set-ups; it performs competitively or better compared to the popular existing methods like K-medoids, TCLUST, trimmed K-means and MCLUST, especially when the mixture components (i.e., the clusters) share regions with significant overlap or outlying clusters exist with small but non-negligible weights (particularly in higher dimensions). Two real datasets are also used to illustrate the performance of the newly proposed method in comparison with others along with an application in image processing. The proposed method detects the clusters with lower misclassification rates and successfully points out the outlying (anomalous) observations from these datasets.

Improved Accuracy Estimation of the Tikhonov Method for Ill-Posed Optimization Problems in Hilbert Space

Improved Accuracy Estimation of the Tikhonov Method for Ill-Posed Optimization Problems in Hilbert Space

Selecting the Regularization Parameter in the Distribution of Relaxation Times

Electrochemical impedance spectroscopy (EIS) is used widely in electrochemistry. Obtaining EIS data is simple with modern electrochemical workstations. Yet, analyzing EIS spectra is still a considerable quandary. The distribution of relaxation times (DRT) has emerged as a solution to this challenge. However, DRT deconvolution underlies an ill-posed optimization problem, often solved by ridge regression, whose accuracy strongly depends on the regularization level This article studies the selection of using several cross-validation (CV) methods and the L-curve approach. A hierarchical Bayesian DRT (hyper-) deconvolution method is also analyzed, whereby a parameter analogous to is obtained through CV. The analysis of a synthetic dataset suggests that the values of selected by generalized and modified generalized CV are the most accurate among those studied. Furthermore, the analysis of synthetic EIS spectra indicates that the hyper- approach outperforms optimal ridge regression. Due to its broad scope, this research will foster additional research on the vital topics of hyperparameter selection for DRT deconvolution. This article also provides, through pyDRTtools, an implementation, which will serve as a starting point for future research.

Gradient Matters: Designing Binarized Neural Networks via Enhanced Information-Flow.

Binarized neural networks (BNNs) have drawn significant attention in recent years, owing to great potential in reducing computation and storage consumption. While it is attractive, traditional BNNs usually suffer from slow convergence speed and dramatical accuracy-degradation on large-scale classification datasets. To minimize the gap between BNNs and deep neural networks (DNNs), we propose a new framework of designing BNNs, dubbed Hyper-BinaryNet, from the aspect of enhanced information-flow. Our contributions are threefold: 1) Considering the capacity-limitation in the backward pass, we propose an 1-bit convolution module named HyperConv. By exploiting the capacity of auxiliary neural networks, BNNs gain better performance on large-scale image classification task. 2) Considering the slow convergence speed in BNNs, we rethink the gradient accumulation mechanism and propose a hyper accumulation technique. By accumulating gradients in multiple variables rather than one as before, the gradient paths for each weight increase, which escapes BNNs from the gradient bottleneck problem during training. 3) Considering the ill-posed optimization problem, a novel gradient estimation warmup strategy, dubbed STE-Warmup, is developed. This strategy prevents BNNs from the unstable optimization process by progressively transferring neural networks from 32-bit to 1-bit. We conduct evaluations with variant architectures on three public datasets: CIFAR-10/100 and ImageNet. Compared with state-of-the-art BNNs, Hyper-BinaryNet shows faster convergence speed and outperforms existing BNNs by a large margin.

Three-Dimensional Reconstruction from a Single RGB Image Using Deep Learning: A Review.

Performing 3D reconstruction from a single 2D input is a challenging problem that is trending in literature. Until recently, it was an ill-posed optimization problem, but with the advent of learning-based methods, the performance of 3D reconstruction has also significantly improved. Infinitely many different 3D objects can be projected onto the same 2D plane, which makes the reconstruction task very difficult. It is even more difficult for objects with complex deformations or no textures. This paper serves as a review of recent literature on 3D reconstruction from a single view, with a focus on deep learning methods from 2018 to 2021. Due to the lack of standard datasets or 3D shape representation methods, it is hard to compare all reviewed methods directly. However, this paper reviews different approaches for reconstructing 3D shapes as depth maps, surface normals, point clouds, and meshes; along with various loss functions and metrics used to train and evaluate these methods.

Defect identification in heat transfer problems using boundary data

The identification of internal defects has been widely focused and employed in the field of nondestructive evaluation (NDE), whereas its application in inverse heat transfer is often restricted by connectedness. Thus, in this paper, a novel identification method of multiple defects based on the parametric level set method (PLSM) and the Method of Moving Asymptotes (MMA) is proposed, where the connectedness limitation is released. In PLSM, the compactly supported radial basis functions (CSRBFs) are utilized to decouple the material related level set function, which endows the PLSM to be capable of forming high curvature portions and capturing local details of the defects. After that, the evolution of guessed level set can be tracked by updating the time-dependent expansion coefficients. Then, a least-square problem is defined to find the optimal coefficients that correspond to the actual defects. To obtain the controllable convergence and stabilize the solving of the ill-posed optimization problem, the MMA is employed to minimize the objective function. Furthermore, no requirement for the mesh reconstruction after the usage of ersatz material model, one also renders the sensitivity prone to compute. The efficiency and accuracy of the proposed method are further validated in several numerical examples by discussing the identification of single and multiple defects.

ROSIE: RObust Sparse ensemble for outlIEr detection and gene selection in cancer omics data

The extraction of novel information from omics data is a challenging task, in particular, since the number of features (e.g. genes) often far exceeds the number of samples. In such a setting, conventional parameter estimation leads to ill-posed optimization problems, and regularization may be required. In addition, outliers can largely impact classification accuracy.Here we introduce ROSIE, an ensemble classification approach, which combines three sparse and robust classification methods for outlier detection and feature selection and further performs a bootstrap-based validity check. Outliers of ROSIE are determined by the rank product test using outlier rankings of all three methods, and important features are selected as features commonly selected by all methods.We apply ROSIE to RNA-Seq data from The Cancer Genome Atlas (TCGA) to classify observations into Triple-Negative Breast Cancer (TNBC) and non-TNBC tissue samples. The pre-processed dataset consists of genes and more than samples. We demonstrate that ROSIE selects important features and outliers in a robust way. Identified outliers are concordant with the distribution of the commonly selected genes by the three methods, and results are in line with other independent studies. Furthermore, we discuss the association of some of the selected genes with the TNBC subtype in other investigations. In summary, ROSIE constitutes a robust and sparse procedure to identify outliers and important genes through binary classification. Our approach is ad hoc applicable to other datasets, fulfilling the overall goal of simultaneously identifying outliers and candidate disease biomarkers to the targeted in therapy research and personalized medicine frameworks.

Stepwise Stochastic Dictionary Adaptation Improves Microstructure Reconstruction with Orientation Distribution Function Fingerprinting.

Fitting of the multicompartment biophysical model of white matter is an ill-posed optimization problem. One approach to make it computationally tractable is through Orientation Distribution Function (ODF) Fingerprinting. However, the accuracy of this method relies solely on ODF dictionary generation mechanisms which either sample the microstructure parameters on a multidimensional grid or draw them randomly with a uniform distribution. In this paper, we propose a stepwise stochastic adaptation mechanism to generate ODF dictionaries tailored specifically to the diffusion-weighted images in hand. The results we obtained on a diffusion phantom and in vivo human brain images show that our reconstructed diffusivities are less noisy and the separation of a free water fraction is more pronounced than for the prior (uniform) distribution of ODF dictionaries.

Принцип Лагранжа и его регуляризация как теоретическая основа устойчивого решения задач оптимального управления и обратных задач

The paper is devoted to the regularization of the classical optimality conditions (COC) — the Lagrange principle and the Pontryagin maximum principle in a convex optimal control problem for a parabolic equation with an operator (pointwise state) equality-constraint at the final time. The problem contains distributed, initial and boundary controls, and the set of its admissible controls is not assumed to be bounded. In the case of a specific form of the quadratic quality functional, it is natural to interpret the problem as the inverse problem of the final observation to find the perturbing effect that caused this observation. The main purpose of regularized COCs is stable generation of minimizing approximate solutions (MAS) in the sense of J. Warga. Regularized COCs are: 1) formulated as existence theorems of the MASs in the original problem with a simultaneous constructive representation of specific MASs; 2) expressed in terms of regular classical Lagrange and Hamilton–Pontryagin functions; 3) are sequential generalizations of the COCs and retain the general structure of the latter; 4) “overcome” the ill-posedness of the COCs, are regularizing algorithms for solving optimization problems, and form the theoretical basis for the stable solving modern meaningful ill-posed optimization and inverse problems.

Ill-Posed Nonlinear Optimization Problems and Uniform Accuracy Estimates of Regularization Methods

We investigate the ill-posed problem of minimizing weakly lower semicontinuous functionals on a convex closed set in a Hilbert space. The functionals to be minimized are available with errors. We prove that a necessary condition for the existence of a regularization procedure with a uniform accuracy estimate on the class of weakly lower semicontinuous functionals is the well-posedness of related optimization problems. We also study regularization methods that provide uniform accuracy estimates, linear with respect to the error level in cost functionals. Under appropriate assumptions on the feasible set, we establish a necessary condition for the existence of such regularizing procedures. In the case of unconstrained problems, the obtained condition reduces to the local strong convexity of functionals under minimization.

A generalized modeling of ill-posed inverse reconstruction of images using a novel data-driven framework

By definition, an instance of image reconstruction often combines denoising, deblurring and enhancing objectively and appears as a fundamental process in visual processing systems which is mathematically an ill-posed inverse optimization problem. It suffers additional complexities because of a non-stationary nature of the image, restricting available methods to produce inconsistent restoration on a variety of image and degradation types. It requires filling this gap by a generic framework of image restoration. Therefore, in this paper, we modeled a novel data-driven framework of generic image reconstruction as a testing benchmark to the emerging application of deep learning. The proposed framework comprises of data engineering, image filtering and a regression neural network for better restoration of many degraded images. We evaluated the effectiveness of the framework on many images degraded by a Gaussian, out-of-focus, motion or airy-pattern blur and random additive white Gaussian noise. The performance appeared better in comparison to a benchmark and state-of-the-art generic image restoration results on several indicators. Furthermore, the framework performed better in comparison to a recent approach to image reconstruction which uses a deep convolutional neural network.

ℓ1-Regularized full-waveform inversion with prior model information based on orthant-wise limited memory quasi-Newton method

Full-waveform inversion (FWI) is an ill-posed optimization problem which is sensitive to noise and initial model. To alleviate the ill-posedness of the problem, regularization techniques are usually adopted. The ℓ1-norm penalty is a robust regularization method that preserves contrasts and edges. The Orthant-Wise Limited-Memory Quasi-Newton (OWL-QN) method extends the widely-used limited-memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) method to the ℓ1-regularized optimization problems and inherits the efficiency of L-BFGS. To take advantage of the ℓ1-regularized method and the prior model information obtained from sonic logs and geological information, we implement OWL-QN algorithm in ℓ1-regularized FWI with prior model information in this paper. Numerical experiments show that this method not only improve the inversion results but also has a strong anti-noise ability.

Necessary and sufficient conditions for power convergence rate of approximations in Tikhonov’s scheme for solving ill-posed optimization problems

We investigate a rate of convergence of estimates for approximations generated by Tikhonov’s scheme for solving ill-posed optimization problems with smooth functionals under a structural nonlinearity condition in a Hilbert space, in the cases of exact and noisy input data. In the noise-free case, we prove that the power source representation of the desired solution is close to a necessary and sufficient condition for the power convergence estimate having the same exponent with respect to the regularization parameter. In the presence of a noise, we give a parameter choice rule that leads for Tikhonov’s scheme to a power accuracy estimate with respect to the noise level.

Design of Biomedical Robots for Phenotype Prediction Problems.

Genomics has been used with varying degrees of success in the context of drug discovery and in defining mechanisms of action for diseases like cancer and neurodegenerative and rare diseases in the quest for orphan drugs. To improve its utility, accuracy, and cost-effectiveness optimization of analytical methods, especially those that translate to clinically relevant outcomes, is critical. Here we define a novel tool for genomic analysis termed a biomedical robot in order to improve phenotype prediction, identifying disease pathogenesis and significantly defining therapeutic targets. Biomedical robot analytics differ from historical methods in that they are based on melding feature selection methods and ensemble learning techniques. The biomedical robot mathematically exploits the structure of the uncertainty space of any classification problem conceived as an ill-posed optimization problem. Given a classifier, there exist different equivalent small-scale genetic signatures that provide similar predictive accuracies. We perform the sensitivity analysis to noise of the biomedical robot concept using synthetic microarrays perturbed by different kinds of noises in expression and class assignment. Finally, we show the application of this concept to the analysis of different diseases, inferring the pathways and the correlation networks. The final aim of a biomedical robot is to improve knowledge discovery and provide decision systems to optimize diagnosis, treatment, and prognosis. This analysis shows that the biomedical robots are robust against different kinds of noises and particularly to a wrong class assignment of the samples. Assessing the uncertainty that is inherent to any phenotype prediction problem is the right way to address this kind of problem.

The split Bregman algorithm applied to PDE-constrained optimization problems with total variation regularization

We derive an efficient solution method for ill-posed PDE-constrained optimization problems with total variation regularization. This regularization technique allows discontinuous solutions, which is desirable in many applications. Our approach is to adapt the split Bregman technique to handle such PDE-constrained optimization problems. This leads to an iterative scheme where we must solve a linear saddle point problem in each iteration. We prove that the spectra of the corresponding saddle point operators are almost contained in three bounded intervals, not containing zero, with a very limited number of isolated eigenvalues. Krylov subspace methods handle such operators very well and thus provide an efficient algorithm. In fact, we can guarantee that the number of iterations needed cannot grow faster than $$O([\ln (\alpha ^{-1})]^2)$$O([ln(ź-1)]2) as $$\alpha \rightarrow 0$$źź0, where $$\alpha $$ź is a small regularization parameter. Moreover, in our numerical experiments we demonstrate that one can expect iteration numbers of order $$O(\ln (\alpha ^{-1}))$$O(ln(ź-1)).

Using multiobjective optimization to map the entropy region

Mapping the structure of the entropy region of at least four jointly distributed random variables is an important open problem. Even partial knowledge about this region has far reaching consequences in other areas in mathematics, like information theory, cryptography, probability theory and combinatorics. Presently, the only known method of exploring the entropy region is, or equivalent to, the one of Zhang and Yeung from 1998. Using some non-trivial properties of the entropy function, their method is transformed to solving high dimensional linear multiobjective optimization problems. Benson’s outer approximation algorithm is a fundamental tool for solving such optimization problems. An improved version of Benson’s algorithm is presented, which requires solving one scalar linear program in each iteration rather than two or three as in previous versions. During the algorithm design, special care is taken for numerical stability. The implemented algorithm is used to verify previous statements about the entropy region, as well as to explore it further. Experimental results demonstrate the viability of the improved Benson’s algorithm for determining the extremal set of medium-sized numerically ill-posed optimization problems. With larger problem sizes, two limitations of Benson’s algorithm is observed: the inefficiency of the scalar LP solver, and the unexpectedly large number of intermediate vertices.

Sourcewise representability conditions and power estimates of convergence rate in Tikhonov’s scheme for solving ill-posed extremal problems

We study the rate of the convergence of approximations generated by the Tikhonov scheme for solving ill-posed optimization problems with smooth functionals given in a general form in a Hilbert space. We establish sourcewise representability conditions which are necessary and sufficient for the convergence of approximations at a power rate. Sufficient conditions are related to the estimate of the discrepancy with respect to the objective functional, while the necessary ones are formulated for the estimate with respect to the argument. We specify certain cases when sufficient and necessary conditions coincide in essence.

Parametric Estimation of Ordinary Differential Equations With Orthogonality Conditions

Differential equations are commonly used to model dynamical deterministic systems in applications. When statistical parameter estimation is required to calibrate theoretical models to data, classical statistical estimators are often confronted to complex and potentially ill-posed optimization problem. As a consequence, alternative estimators to classical parametric estimators are needed for obtaining reliable estimates. We propose a gradient matching approach for the estimation of parametric Ordinary Differential Equations (ODE) observed with noise. Starting from a nonparametric proxy of a true solution of the ODE, we build a parametric estimator based on a variational characterization of the solution. As a Generalized Moment Estimator, our estimator must satisfy a set of orthogonal conditions that are solved in the least squares sense. Despite the use of a nonparametric estimator, we prove the - consistency and asymptotic normality of the Orthogonal Conditions estimator. We can derive confidence sets thanks to a closed-form expression for the asymptotic variance. Finally, the OC estimator is compared to classical estimators in several (simulated and real) experiments and ODE models to show its versatility and relevance with respect to classical Gradient Matching and Nonlinear Least Squares estimators. In particular, we show on a real dataset of influenza infection that the approach gives reliable estimates. Moreover, we show that our approach can deal directly with more elaborated models such as Delay Differential Equation (DDE). Supplementary materials for this article are available online.

On the application of the regularization method for the correction of improper problems of convex programming

The residual method, which is one of the standard regularization procedures for ill-posed optimization problems, is applied to a convex programming problem. The connection between this method and the regularized Lagrange function method is investigated in the case of optimal correction of improper problems of convex programming. This approach allows one to decrease the number of impropriety classes to be analyzed. Conditions are formulated and convergence estimates of the method are established.