Least-squares Problem Research Articles

Forward-partial inverse-half-forward splitting algorithm for solving monotone inclusions

In this paper we provide a splitting algorithm for solving coupled monotone inclusions in a real Hilbert space involving the sum of a normal cone to a vector subspace, a maximally monotone, a monotone-Lipschitzian, and a cocoercive operator. The proposed method takes advantage of the intrinsic properties of each operator and generalizes the method of partial inverses and the forward-backward-half forward splitting, among other methods. At each iteration, our algorithm needs two computations of the Lipschitzian operator while the cocoercive operator is activated only once. By using product space techniques, we derive a method for solving a composite monotone primal-dual inclusions including linear operators and we apply it to solve constrained composite convex optimization problems. Finally, we apply our algorithm to a constrained total variation least-squares problem and we compare its performance with efficient methods in the literature.

Improving reconstruction-based coding methods for image classification: a visual dictionary refining method

The reconstruction-based coding methods utilize a dictionary to approximate an input feature as a sparse representation in image classification tasks by solving a least-squares problem with constraints. The input feature can be a handcrafted feature calculated over a small patch or a deep feature extracted by a pre-trained convolutional neural network model. As known, the contribution degrees of features extracted within different regions are distinct to the same classification task. Given this, we argue that for features with a large contribution degree, if their reconstruction errors induced by a reconstruction-based coding method can be decreased, more discriminative information will be retained to improve classification accuracy. Considering that different coding methods have the same lower bound of reconstruction error, we propose a visual dictionary refining method that adjusts only the lower bounds of the errors of features according to their contribution degrees. Concretely, given an initial dictionary used by a reconstruction-based coding method, the indexes (coding indexes) of the words from the dictionary for encoding each feature are obtained. Then, the initial dictionary is updated according to the contribution degrees and coding indexes of features. The highlight of our method is that it can be applied to many reconstruction-based coding methods without damaging their original constraints. Experimental results on 15-Scenes, Caltech-101, and UIUC-Sports show that the classification accuracies of four representative coding methods are improved by 0.17% to 1.5% after applying our method. These results exhibit the effectiveness and universality of our method.

A Least-Squares Method for the Solution of the Non-smooth Prescribed Jacobian Equation

We consider a least-squares/relaxation finite element method for the numerical solution of the prescribed Jacobian equation. We look for its solution via a least-squares approach. We introduce a relaxation algorithm that decouples this least-squares problem into a sequence of local nonlinear problems and variational linear problems. We develop dedicated solvers for the algebraic problems based on Newton’s method and we solve the differential problems using mixed low-order finite elements. Various numerical experiments demonstrate the accuracy, efficiency and the robustness of the proposed method, compared for instance to augmented Lagrangian approaches.

SLAM on the Hexagonal Grid.

Hexagonal grids have many advantages over square grids and could be successfully used in mobile robotics as a map representation. However, there is a lack of an essential algorithm, namely, SLAM (simultaneous localization and mapping), that would generate a map directly on the hexagonal grid. In this paper, this issue is addressed. The solution is based on scan matching and solving the least-square problem with the Gauss–Newton formula, but it is modified with the Lagrange multiplier theorem. This is necessary to fulfill the constraints given by the manifold. The algorithm was tested in the synthetic environment and on a real robot and is entirely fully suitable for the presented task. It generates a very accurate map and generally has even better precision than the similar approach implemented on the square lattice.

Numerical solution of density-driven groundwater flows using a generalized finite difference method defined by an unweighted least-squares problem

Density-driven groundwater flows are described by nonlinear coupled differential equations. Due to its importance in engineering and earth science, several linearizations and semi-linearization schemes for approximating their solution have been proposed. Among the more efficient are the combinations of Newtonian iterations for the spatially discretized system obtained by either scalar homotopy methods, fictitious time methods, or meshless generalized finite difference method, with several implicit methods for the time integration. However, when these methods are used, some parameters need to be determined, in some cases, even manually. To overcome this problem, this paper presents a novel generalized finite differences scheme combined with an adaptive step-size method, which can be applied for solving the governing equations of interest on non-rectangular structured and unstructured grids. The proposed method is tested on the Henry and the Elder problems to verify the accuracy and the stability of the proposed numerical scheme.

Algebraic methods for equality constrained least squares problems in commutative quaternionic theory

This paper, by means of two matrix representations of a commutative quaternion matrix, studies the relationship between the solutions of commutative quaternion equality constrained least squares (LSE) problems and that of complex and real LSE problems and derives two algebraic methods for finding the solutions of equality constrained least squares problems in commutative quaternionic theory.

Decentralized proximal splitting algorithms for composite constrained convex optimization

This paper concentrates on a class of decentralized convex optimization problems subject to local feasible sets, equality and inequality constraints, where the global objective function consists of a sum of locally smooth convex functions and non-smooth regularization terms. To address this problem, a synchronous full-decentralized primal-dual proximal splitting algorithm (Syn-FdPdPs) is presented, which avoids the unapproximable property of the proximal operator with respect to inequality constraints via logarithmic barrier functions. Following the proposed decentralized protocol, each agent carries out local information exchange without any global coordination and weight balancing strategies introduced in most consensus algorithms. In addition, a randomized version of the proposed algorithm (Rand-FdPdPs) is conducted through subsets of activated agents, which further removes the global clock coordinator. Theoretically, with the help of asymmetric forward-backward-adjoint (AFBA) splitting technique, the convergence results of the proposed algorithms are provided under the same local step-size conditions. Finally, the effectiveness and practicability of the proposed algorithms are demonstrated by numerical simulations on the least-square and least absolute deviation problems.

K-Best Sphere Decoding Algorithm for Long Prediction Horizon FCS-MPC

Finite control set model predictive control (FCS-MPC) strategies for power conversion devices benefit from extending the prediction horizon length. Solving this problem relies on the definition of the underlying integer least-squares problem. Sphere decoding algorithm (SDA) has been extensively used in previous works as an approach to solve this problem. In this article, a parallel and fully scalable K-best SDA hardware design is proposed as an alternative. The K-best SDA establishes a different breadth-first search strategy, which addresses some of the main drawbacks of the SDA. Through experimental tests based on an uninterruptible power supply, the K-best SDA performance for long prediction horizon FCS-MPC is assessed and verified. Results demonstrate key beneficial aspects through which the K-best SDA is capable of rendering an improved control performance when compared to the conventional SDA.

An improved iterative model reduction technique to estimate the unknown responses using limited available responses

A major challenge in structural health monitoring (SHM) is the availability of responses at limited degrees of freedom (DOFs). This requires the determination of the transformation parameter to expand the available DOFs to the full size of a model. The present study proposes an improved iterative model reduction algorithm by eliminating the stiffness terms from the transformation equation. The modified equation is a function of measured modal responses and mass matrices. This enables obtaining the unknown responses without repeated evaluations in case of stiffness reductions typically involved in SHM problems. The transformation parameter is solved using an improved Levenberg-Marquardt (LM) based iterative least-squares optimization technique. Specifically, the LM algorithm is enhanced by introducing an adaptive damping term in the least-squares problem. The proposed approach is further integrated into the substructuring scheme so that it can be readily applied for large finite element models. The algorithm is numerically demonstrated by considering a beam and a ten-storey building model using modal data with Gaussian noise. The effectiveness of the proposed reduced-order model is studied for a gradually decreasing number of modes and available responses for various measurement configurations by comparing with the results of the existing model reduction techniques.

A graph-space optimal transport objective function based on q-statistics to mitigate cycle-skipping issues in FWI

SUMMARY Full-waveform inversion (FWI) is a powerful seismic imaging methodology to estimate geophysical parameters that honour the recorded waveforms (observed data), and it is conventionally formulated as a least-squares optimization problem. Despite many successful applications, least-squares FWI suffers from cycle skipping issues. Optimal transport (OT) based FWI has been demonstrated to be a useful strategy for mitigating cycle skipping. In this work, we introduce a new Wasserstein metric based on q-statistics in the context of the OT distance. In this sense, instead of the data themselves, we consider the graph of the seismic data, which are positive and normalized quantities similar to probability functions. By assuming that the difference between the graphs of the modelled and observed data obeys the q-statistics, we introduce a robust q-generalized graph-space OT objective function in the FWI context namely q-GSOT-FWI, in which the standard GSOT-FWI based on l2-norm is a particular case. To demonstrate how the q-GSOT-FWI deals with cycle skipping, we present two numerical examples involving 2-D acoustic wave-equation modelling. First, we investigate the convexity of q-GSOT objective function regarding different time-shifts, and, secondly, we present a Brazilian pre-salt synthetic case study, from a crude initial model which generates significant cycle-skipping seismic data. The results reveal that the q-GSOT-FWI is a powerful strategy to circumvent cycle skipping issues in FWI, in which our objective function proposal presents a smoother topography with a wider attraction valley to the optimal minimum. They also show that q-statistics leads to a significant improvement of FWI objective function convergence, generating higher resolution acoustic models than classical approaches. In addition, our proposal reduces the computational cost of calculating the transport plan as the q-value increases.

Fast and efficient PAT image reconstruction algorithms: A comparative performance analysis

Medical imaging technology efficiency is determined by how accurate, rapid, invasive, or non-invasive it is. Photoacoustic tomography (PAT) is a noninvasive emerging technology for medical imaging that is able to produce images with high resolution and high contrast in long penetration depths. In PAT, image reconstruction accuracy relies on the accuracy of the system model. Accurate model-based methods can be made use of for efficiently reconstructing a PAT image from the sensor measurements using a pseudo-spectral method which is suggested to produce exactly the same results as that of the k-Wave toolbox in MATLAB. Compressive sensing has been successfully applied for medical image reconstruction as it allows for sparse sampling and sparse reconstruction. Compressive sensing combined with PAT allows for deep tissue imaging at higher resolution at a much lower computational time. A fast and efficient PAT image reconstruction compressive sensing algorithm is needed to be developed, which can image dense tissues (such as breast tissue) rapidly, for early detection of cancer. In this paper, relaxed- Basis pursuit using alternating direction method of multipliers and split Bregman formulation of the total variation regularized least-squares problem is proposed and is observed to work under practical scenarios with fewer samples and sensors. Moreover, a comparative study for the performance analysis of the different compressive sensing algorithms, both iterative and non-iterative, is carried out for a decreased number of sensors and samples. The proposed split Bregman total variation algorithm is fast and produces a high-resolution sparsity preserving image compared to the state-of-the-art algorithms. Moreover, for a reduced number of samples, the proposed relaxed-Basis Pursuit is desired and performs much better than all the algorithms discussed. Implementing Cholesky factorization to the proposed algorithms further reduces the computation time of the split-Bregman algorithm by 90%. The new methods outperform existing ones in practical circumstances recreating a high-resolution image, even when the number of sensors and samples is reduced.

PERF: Performant, Explicit Radiance Fields

We present a novel way of approaching image-based 3D reconstruction based on radiance fields. The problem of volumetric reconstruction is formulated as a non-linear least-squares problem and solved explicitly without the use of neural networks. This enables the use of solvers with a higher rate of convergence than what is typically used for neural networks, and fewer iterations are required until convergence. The volume is represented using a grid of voxels, with the scene surrounded by a hierarchy of environment maps. This makes it possible to get clean reconstructions of 360° scenes where the foreground and background is separated. A number of synthetic and real scenes from well-known benchmark-suites are successfully reconstructed with quality on par with state-of-the-art methods, but at significantly reduced reconstruction times.

An extended primal-dual algorithm framework for nonconvex problems: application to image reconstruction in spectral CT

Using the convexity of each component of the forward operator, we propose an extended primal-dual algorithm framework for solving a kind of nonconvex and probably nonsmooth optimization problems in spectral computed tomography (CT) image reconstruction. Following the proposed algorithm framework, we present six different iterative schemes or algorithms, and then establish the relationship to some existing algorithms. Under appropriate conditions, we prove the convergence of these schemes for the general case. Moreover, when the proposed schemes are applied to solving a specific problem in spectral CT image reconstruction, namely, total variation regularized nonlinear least-squares problem with nonnegative constraint, we also prove the particular convergence for these schemes by using some special properties. The numerical experiments with densely and sparsely data demonstrate the convergence and accuracy of the proposed algorithm framework in terms of visual inspection of images of realistic anatomic complexity and quantitative analysis with metrics structural similarity, peak signal-to-noise ratio, mean square error and maximum pixel difference. We analyze the computational complexity of these schemes, and discuss the extended applications of this algorithm framework in other nonlinear imaging problems.

Hybrid Interval-Probabilistic Localization in Building Maps

We present a novel online capable hybrid interval-probabilistic localization method using publicly available 2D building maps. Given an initially large uncertainty for the orientation and position derived from GNSS data, our novel interval-based approach first narrows down the orientation to a smaller interval and provides a set described by a minimal polygon for the position of the vehicle that encloses the feasible set of poses by taking the building geometry into account using 3D Light Detection and Ranging (LiDAR) sensor data. Second, we perform a probabilistic Maximum Likelihood Estimation (MLE) to determine the best solution within the determined feasible set. The MLE is converted into a least-squares problem that is solved by an optimization approach that takes the bounds of the solution set into account so that only a solution within the feasible set is selected as the most likely one. We experimentally show with real data that the novel interval-based localization provides sets of poses that contain the true pose for more than 99% of the frames and that the bounded optimization provides more reliable results compared to a classical unbounded optimization and a Monte Carlo Localization approach.

A Contraction Analysis of Primal-Dual Dynamics in Distributed and Time-Varying Implementations

In this article, we provide an overarching analysis of primal-dual dynamics associated with linear equality-constrained optimization problems using contraction analysis. For the well-known standard version of the problem, we establish convergence under convexity and the contracting rate under strong convexity. Then, for a canonical distributed optimization problem, we use partial contractivity to establish global exponential convergence of its primal-dual dynamics. As an application, we propose a new distributed solver for the least-squares problem with the same convergence guarantees. Finally, for time-varying versions of both centralized and distributed primal-dual dynamics, we exploit their contractive nature to establish bounds on their tracking error. To support our analyses, we introduce novel results on contraction theory.

Scaling multi-instance support vector machine to breast cancer detection on the BreaKHis dataset.

MotivationBreast cancer is a type of cancer that develops in breast tissues, and, after skin cancer, it is the most commonly diagnosed cancer in women in the United States. Given that an early diagnosis is imperative to prevent breast cancer progression, many machine learning models have been developed in recent years to automate the histopathological classification of the different types of carcinomas. However, many of them are not scalable to large-scale datasets.ResultsIn this study, we propose the novel Primal-Dual Multi-Instance Support Vector Machine to determine which tissue segments in an image exhibit an indication of an abnormality. We derive an efficient optimization algorithm for the proposed objective by bypassing the quadratic programming and least-squares problems, which are commonly employed to optimize Support Vector Machine models. The proposed method is computationally efficient, thereby it is scalable to large-scale datasets. We applied our method to the public BreaKHis dataset and achieved promising prediction performance and scalability for histopathological classification.Availability and implementationSoftware is publicly available at: https://1drv.ms/u/s!AiFpD21bgf2wgRLbQq08ixD0SgRD?e=OpqEmY.Supplementary information Supplementary data are available at Bioinformatics online.

Parameter Identification Method for Nonsmooth Aeroelastic System

In this paper, a parameter identification method merged with the precise integration method and the predictor–corrector strategy is proposed to identify the parameters of a nonsmooth aeroelastic system quickly and accurately. Considering the aeroelastic system subject to unsteady aerodynamic force, the parameters that need to be identified include the structural and nonsmooth parameters. The work of this paper mainly includes three parts. First, based on the precise integration method and combined with the predictor–corrector strategy, a new technology that can accurately locate the switching point is proposed to obtain the response of the aeroelastic system with high precision. Then, the problem for identification of parameters of the nonsmooth aeroelastic system is modeled as a nonlinear least-squares problem between the measured data and the calculated response. A new strategy that can obtain the nonsmooth parameters sensitivity matrix accurately is proposed, and then the parameters are identified through the enhanced response sensitivity approach. Finally, the nonsmooth aeroelastic system with freeplay and hysteresis nonlinearity are taken as the numerical examples to demonstrate the feasibility and robustness of the present method.

Scalable subspace methods for derivative-free nonlinear least-squares optimization

We introduce a general framework for large-scale model-based derivative-free optimization based on iterative minimization within random subspaces. We present a probabilistic worst-case complexity analysis for our method, where in particular we prove high-probability bounds on the number of iterations before a given optimality is achieved. This framework is specialized to nonlinear least-squares problems, with a model-based framework based on the Gauss–Newton method. This method achieves scalability by constructing local linear interpolation models to approximate the Jacobian, and computes new steps at each iteration in a subspace with user-determined dimension. We then describe a practical implementation of this framework, which we call DFBGN. We outline efficient techniques for selecting the interpolation points and search subspace, yielding an implementation that has a low per-iteration linear algebra cost (linear in the problem dimension) while also achieving fast objective decrease as measured by evaluations. Extensive numerical results demonstrate that DFBGN has improved scalability, yielding strong performance on large-scale nonlinear least-squares problems.

Grouped Transformations and Regularization in High-Dimensional Explainable ANOVA Approximation

In this paper we propose a tool for high-dimensional approximation based on trigonometric polynomials where we allow only low-dimensional interactions of variables. In a general high-dimensional setting, it is already possible to deal with special sampling sets such as sparse grids or rank-1 lattices. This requires black-box access to the function, i.e., the ability to evaluate it at any point. Here, we focus on scattered data points and grouped frequency index sets along the dimensions. From there we propose a fast matrix-vector multiplication, the grouped Fourier transform, for high-dimensional grouped index sets. Those transformations can be used in the application of the previously introduced method of approximating functions with low superposition dimension based on the analysis of variance (ANOVA) decomposition where there is a one-to-one correspondence from the ANOVA terms to our proposed groups. The method is able to dynamically detected important sets of ANOVA terms in the approximation. In this paper, we consider the involved least-squares problem and add different forms of regularization: Classical Tikhonov-regularization, namely, regularized least squares and the technique of group lasso, which promotes sparsity in the groups. As for the latter, there are no explicit solution formulas which is why we applied the fast iterative shrinking-thresholding algorithm to obtain the minimizer. Moreover, we discuss the possibility of incorporating smoothness information into the least-squares problem. Numerical experiments in under-, overdetermined, and noisy settings indicate the applicability of our algorithms. While we consider periodic functions, the idea can be directly generalized to non-periodic functions as well.

Optimization landscape in the simplest constrained random least-square problem

We analyze statistical features of the ‘optimization landscape’ in a random version of one of the simplest constrained optimization problems of the least-square type: finding the best approximation for the solution of a system of M linear equations in N unknowns: ( a k , x ) = b k , k = 1, …, M on the N-sphere x 2 = N. We treat both the N-component vectors a k and parameters b k as independent mean zero real Gaussian random variables. First, we derive the exact expressions for the mean number of stationary points of the least-square loss function in the overcomplete case M > N in the framework of the Kac-Rice approach combined with the random matrix theory for Wishart ensemble. Then we perform its asymptotic analysis as N → ∞ at a fixed α = M/N > 1 in various regimes. In particular, this analysis allows to extract the large deviation function for the density of the smallest Lagrange multiplier λ min associated with the problem, and in this way to find its most probable value. This can be further used to predict the asymptotic mean minimal value of the loss function as N → ∞. Finally, we develop an alternative approach based on the replica trick to conjecture the form of the large deviation function for the density of at N ≫ 1 and any fixed ratio α = M/N > 0. As a by-product, we find the compatibility threshold α c < 1 which is the value of α beyond which a large random linear system on the N-sphere becomes typically incompatible.