Least-squares Problem Research Articles

Tightly coupled stereo vision-inertial odometry based on point and line feature

Abstract To improve the accuracy and robustness of visual simultaneous localization and mapping (SLAM) in low-texture environments, this paper proposes a robust and fast stereo vision inertial SLAM pose estimation method that combines point and line features with an inertial measurement unit (IMU). The method tightly couples visual point and line features with IMU constraints, forming a least-squares problem through the error of each constraint term for nonlinear optimization. To address the issues of over-segmentation and time consumption in traditional line segment detection (LSD) algorithms, an improved LSD algorithm is adopted to accelerate line feature extraction. This approach merges nearby line segments based on spatial geometric relationships and filters out invalid segments, improving the time efficiency of the algorithm. Finally, experiments conducted in low-texture environments demonstrate that our algorithm achieves high localization accuracy and robustness.

Fast and Forward Stable Randomized Algorithms for Linear Least-Squares Problems

Fast and Forward Stable Randomized Algorithms for Linear Least-Squares Problems

Revisiting the fitting of the Nelson–Siegel and Svensson models

The Nelson–Siegel and the Svensson models are two widely used models for the term structure of interest rates. These models are quite simple and intuitive, but fitting them to market data is numerically challenging and various difficulties have been reported. In this paper, we provide a novel mathematical analysis of the fitting problem based on parametric optimization. We formulate the fitting problem as a separable nonlinear least-squares problem, in which the linear parameters can be eliminated. We provide a thorough discussion on the conditioning of the inner part of the reformulated problem and show that many of the reported difficulties encountered when solving it are inherent to the problem formulation itself and cannot be tackled by choosing a particular optimization algorithm. Our stability analysis provides novel insights that we use to show that some of the ill-conditioning can be avoided, and that a suitably chosen penalty approach can be used to address the remaining ill-conditioning. Numerical results indicate that this approach has the expected impact while being independent of any choice of a particular optimization algorithm. We further establish smoothness properties of the reduced objective function, putting global optimization methods for the reduced problem on a sound mathematical basis.

Real-Time Aerodynamic Load Estimation for Hypersonics via Strain-Based Inverse Maps

This work develops an efficient real-time inverse formulation for inferring the aerodynamic surface pressures on a hypersonic vehicle from sparse measurements of the structural strain. The approach aims to provide real-time estimates of the aerodynamic loads acting on the vehicle for ground and flight testing, as well as guidance, navigation, and control applications. Specifically, the approach targets hypersonic flight conditions where direct measurement of the surface pressures is challenging due to the harsh aerothermal environment. For problems employing a linear elastic structural model, the inference problem can be posed as a least-squares problem with a linear constraint arising from a finite element discretization of the governing elasticity partial differential equation. Due to the linearity of the problem, an explicit solution is given by the normal equations. Precomputation of the resulting inverse map enables rapid evaluation of the surface pressure and corresponding integrated quantities, such as the force and moment coefficients. The inverse approach additionally allows for uncertainty quantification, providing insights for theoretical recoverability and robustness to sensor noise. Numerical studies demonstrate the estimator performance for reconstructing the surface pressure field, as well as the force and moment coefficients, for the Initial Concept 3.X (IC3X) conceptual hypersonic vehicle.

Survey of a class of iterative row-action methods: The Kaczmarz method

AbstractThe Kaczmarz algorithm is an iterative method that solves linear systems of equations. It stands out among iterative algorithms when dealing with large systems for two reasons. First, at each iteration, the Kaczmarz algorithm uses a single equation, resulting in minimal computational work per iteration. Second, solving the entire system may only require the use of a small subset of the equations. These characteristics have attracted significant attention to the Kaczmarz algorithm. Researchers have observed that randomly choosing equations can improve the convergence rate of the algorithm. This insight led to the development of the Randomized Kaczmarz algorithm and, subsequently, several other variations emerged. In this paper, we extensively analyze the native Kaczmarz algorithm and many of its variations using large-scale systems as benchmarks. Through our investigation, we have verified that, for consistent systems, various row sampling schemes can outperform both the original and Randomized Kaczmarz method. Specifically, sampling without replacement and using quasirandom numbers are the fastest techniques. However, for inconsistent systems, the Conjugate Gradient method for Least-Squares problems overcomes all variations of the Kaczmarz method for these types of systems.

Multigrid Preconditioning for Regularized Least-Squares Problems

Multigrid Preconditioning for Regularized Least-Squares Problems

Another look at residual dynamic mode decomposition in the regime of fewer snapshots than dictionary size

Residual Dynamic Mode Decomposition (ResDMD) offers a method for accurately computing the spectral properties of Koopman operators. It achieves this by calculating an infinite-dimensional residual from snapshot data, thus overcoming issues associated with finite truncations of Koopman operators (e.g., Extended Dynamic Mode Decomposition), such as spurious eigenvalues. Spectral properties computed by ResDMD include spectra, pseudospectra, spectral measures, Koopman mode decompositions, and dictionary verification. In scenarios where the number of snapshots is fewer than the dictionary size, particularly for exact DMD and kernelized Extended DMD, ResDMD has traditionally been applied by dividing snapshot data into a training set and a quadrature set. We demonstrate how to eliminate the need for two datasets through a novel computational approach of solving a dual least-squares problem. We analyze these new residuals for exact DMD and kernelized Extended DMD, demonstrating ResDMD’s versatility and broad applicability across various dynamical systems, including those modeled by high-dimensional and nonlinear observables. The utility of these new residuals is showcased through three diverse examples: the analysis of a cylinder wake, the study of airfoil cascades, and the compression of transient shockwave experimental data. This approach not only simplifies the application of ResDMD but also extends its potential for deeper insights into the dynamics of complex systems.

Convergent least-squares optimization methods for variational data assimilation

Data assimilation combines prior (or background) information with observations to estimate the initial state of a dynamical system over a given time-window. A common application is in numerical weather prediction where a previous forecast and atmospheric observations are used to obtain the initial conditions for a numerical weather forecast. In four-dimensional variational data assimilation (4D-Var), the problem is formulated as a nonlinear least-squares problem, usually solved using a variant of the classical Gauss-Newton (GN) method. However, we show that GN may not converge if poorly initialized. In particular, we show that this may occur when there is greater uncertainty in the background information compared to the observations, or when a long time-window is used in 4D-Var allowing more observations. The difficulties GN encounters may lead to inaccurate initial state conditions for subsequent forecasts. To overcome this, we apply two convergent GN variants (line search and regularization) to the long time-window 4D-Var problem and investigate the cases where they locate a more accurate estimate compared to GN within a given budget of computational time and cost. We show that these methods are able to improve the estimate of the initial state, which may lead to a more accurate forecast. Highlights Poor initialization of Gauss-Newton method may result in failure to converge. Safeguarded Gauss-Newton improves initial state estimate within limited time/cost. Results using twin experiments with long time-window and chaotic Lorenz models. Apply state of the art least-squares convergence theory to data assimilation. Improvements to initial state estimate may lead to a more accurate forecast.

A Kaczmarz-inspired approach to accelerate the optimization of neural network wavefunctions

Neural network wavefunctions optimized using the variational Monte Carlo method have been shown to produce highly accurate results for the electronic structure of atoms and small molecules, but the high cost of optimizing such wavefunctions prevents their application to larger systems. We propose the Subsampled Projected-Increment Natural Gradient Descent (SPRING) optimizer to reduce this bottleneck. SPRING combines ideas from the recently introduced minimum-step stochastic reconfiguration optimizer (MinSR) and the classical randomized Kaczmarz method for solving linear least-squares problems. We demonstrate that SPRING outperforms both MinSR and the popular Kronecker-Factored Approximate Curvature method (KFAC) across a number of small atoms and molecules, given that the learning rates of all methods are optimally tuned. For example, on the oxygen atom, SPRING attains chemical accuracy after forty thousand training iterations, whereas both MinSR and KFAC fail to do so even after one hundred thousand iterations.

A Fixed-Time Noise-Tolerance ZNN Model for Time-Variant Inequality-Constrained Quaternion Matrix Least-Squares Problem.

Presently, numerical algorithms for solving quaternion least-squares problems have been intensively studied and utilized in various disciplines. However, they are unsuitable for solving the corresponding time-variant problems, and thus few studies have explored the solution to the time-variant inequality-constrained quaternion matrix least-squares problem (TVIQLS). To do so, this article designs a fixed-time noise-tolerance zeroing neural network (FTNTZNN) model to determine the solution of the TVIQLS in a complex environment by exploiting the integral structure and the improved activation function (AF). The FTNTZNN model is immune to the effects of initial values and external noise, which is much superior to the conventional zeroing neural network (CZNN) models. Besides, detailed theoretical derivations about the global stability, the fixed-time (FXT) convergence, and the robustness of the FTNTZNN model are provided. Simulation results indicate that the FTNTZNN model has a shorter convergence time and superior robustness compared to other zeroing neural network (ZNN) models activated by ordinary AFs. At last, the construction method of the FTNTZNN model is successfully applied to the synchronization of Lorenz chaotic systems (LCSs), which shows the practical application value of the FTNTZNN model.

A Recurrent Stochastic Configuration Network for Dynamic System Modeling and Its Application

In order to model nonlinear dynamic systems quickly and accurately, a recurrent stochastic configuration network (RSCN) with universal approximation ability is proposed in this paper. The method consists of network parameter learning and network structure determination. The network parameter learning strategy consists of input weights and the biases of hidden nodes generated randomly through a supervision mechanism, while the output weights of hidden nodes are determined by solving the least-squares problem. The network structure is determined incrementally by the training error of a single-layer recurrent neural network. The effectiveness of RSCN is tested and evaluated by using three nonlinear dynamic functions and historical data from a municipal solid waste incineration (MSWI) process. The experimental results show that the proposed method can improve modeling accuracy and reduce training time, which is valuable for applications in the field of industrial process modeling.

Image up-sampling method for dual-radiation nondestructive cargo inspection system

In this paper, an image up-sampling method for a dual-radiation nondestructive cargo inspection system, that uses x-ray and neutron, is proposed. The inspection system is assumed to have a larger neutron detector pixel size than that of x-ray detector pixel, which is usually the case to address the lower neutron source flux and the lower detector efficiency than those in the x-ray imaging system. The proposed technique adopts a cluster map of an x-ray projection image with high spatial resolution to up-sample the corresponding neutron projection image with low spatial resolution. The cluster map is acquired by use of a modified k-means clustering algorithm. The up-sampling process is formulated by a linear least-square problem of mapping the neutron projection image onto the cluster map. Consequently, the neutron projection image would have the same dimension with the x-ray projection image array allowing R-value based material decomposition. We have conducted a numerical study using GATE simulation to demonstrate the efficacy of the proposed method. The proposed method was compared with a linear interpolation method and also with an up-sampling technique that does not use the cluster map. The outperformance of the proposed method over those reference techniques has been successfully presented.

Stabilization of SIMPLE-like RANS solvers for computing accurate gradients using the complex-step derivative method

The Reynolds-averaged Navier-Stokes (RANS) approach in Computational Fluid Dynamics (CFD) is increasingly vital for aerodynamic design across wind turbines, gas turbines, aircraft, and rotorcraft. Enhancing the process through CFD-based design optimization requires numerous design variables, favoring gradient-based methods for better scalability than gradient-free methods. This research uses numerical differentiation techniques, particularly the complex-step derivative method, to compute gradients. Challenges arise during aerodynamic shape optimization when unconventional shapes disrupt RANS solver assumptions, causing convergence failures. These failures undermine optimization, prompting the need to enhance solver convergence for robust optimization. The modified-Boostconv method is a residual recombination method bolstering unstable eigenvalues to stabilize convergence of iterative solvers for nonlinear systems of equations. This study extends the modified-Boostconv method to combine it with the complex-step derivative technique, creating robust optimizations via RANS-CFD with accurate gradients, even for these numerically unstable cases. The main issue encountered during the complexification of the modified-Boostconv method is how to correctly ensure that the model reduction leading to the least-squares problem satisfies the complex-step derivative method. We test whether the dot product operation involved in the model reduction should be a Hermitian or non-Hermitian inner product. The problem is first tested for a simple analytical case using the logistic equation in combination with the modified-Boostconv method. It shows that the non-Hermitian inner product should be used. This is also confirmed with a similar study using the RANS solver.

Moving horizon estimation for localization of mobile robots with measurement outliers

This paper investigates the moving horizon estimation (MHE) problem of mobile robots with measurement outliers. To deal with measurement outliers, the Euclidean distance of measurement error is introduced to detect and remove abnormal data. Then, we use dimension expansion methods to preprocess the data of heterogeneous sensors, such as UWB and IMU. An MHE-based method is proposed that deals with the localization of mobile robots with measurement outliers in the presence of bounded noise. An MHE-based estimator is obtained by solving a regularized least-squares problem. We analyze the convergence of the estimation error system using the properties of norm inequalities, and an upper bound is derived for the estimation error system by using norm inequality. Finally, simulation and experiment examples are given to verify the effectiveness and applicability of the proposed approach.

Solving Least-Squares Problems via a Double-Optimal Algorithm and a Variant of the Karush–Kuhn–Tucker Equation for Over-Determined Systems

A double optimal solution (DOS) of a least-squares problem Ax=b,A∈Rq×n with q≠n is derived in an m+1-dimensional varying affine Krylov subspace (VAKS); two minimization techniques exactly determine the m+1 expansion coefficients of the solution x in the VAKS. The minimal-norm solution can be obtained automatically regardless of whether the linear system is consistent or inconsistent. A new double optimal algorithm (DOA) is created; it is sufficiently time saving by inverting an m×m positive definite matrix at each iteration step, where m≪min(n,q). The properties of the DOA are investigated and the estimation of residual error is provided. The residual norms are proven to be strictly decreasing in the iterations; hence, the DOA is absolutely convergent. Numerical tests reveal the efficiency of the DOA for solving least-squares problems. The DOA is applicable to least-squares problems regardless of whether q<n or q>n. The Moore–Penrose inverse matrix is also addressed by adopting the DOA; the accuracy and efficiency of the proposed method are proven. The m+1-dimensional VAKS is different from the traditional m-dimensional affine Krylov subspace used in the conjugate gradient (CG)-type iterative algorithms CGNR (or CGLS) and CGRE (or Craig method) for solving least-squares problems with q>n. We propose a variant of the Karush–Kuhn–Tucker equation, and then we apply the partial pivoting Gaussian elimination method to solve the variant, which is better than the original Karush–Kuhn–Tucker equation, the CGNR and the CGNE for solving over-determined linear systems. Our main contribution is developing a double-optimization-based iterative algorithm in a varying affine Krylov subspace for effectively and accurately solving least-squares problems, even for a dense and ill-conditioned matrix A with q≪n or q≫n.

Locating clustered seismicity using Distance Geometry Solvers: applications for sparse and single-borehole DAS networks

Summary The determination of seismic event locations with sparse networks or single-borehole systems remains a significant challenge in observational seismology. Leveraging the advantages of the location approach HADES (eartHquake locAtion via Distance gEometry Solvers), which was initially developed for locating clustered seismicity recorded at two stations, through the solution of a Distance Geometry Problem, we present here an improved version of the methodology: HADES-R (HADES-Relative). Where HADES previously needed a minimum of 4 absolutely located master events, HADES-R solves a least-squares problem to find the relative inter-event distances in the cluster, and uses only a single master event to find the locations of all events, and subsequently applies rotational optimizer to find the cluster orientation. It can leverage iterative station combinations if multiple receivers are available, to describe the cluster shape and orientation uncertainty with a bootstrap approach. The improved method requires P- and S-phase arrival picks, a homogeneous velocity model, a single master event with a known location, and an estimate of the cluster width. The approach is benchmarked on the 2019 Ridgecrest sequence recorded at two stations, and applied to two seismic clusters at the FORGE geothermal test site in Utah, USA, with a microseismic monitoring scenario with a DAS in a vertical borehole. Traditional procedures struggle in these settings due to the ill-posed network configuration. The azimuthal ambiguity in such a scenario is partially overcome by the assumption that all events belong to the same cluster around the master event and a cluster width estimate. We are able to find the cluster shape in both cases, although the orientation remains uncertain. HADES-R contributes to an efficient way to locate multiple events simultaneously with minimal prior information. The method’s ability to constrain the cluster shape and location with only one well-located event offers promising implications, especially for environments where limited or specialised instrumentation is in use.

Simultaneous geometric calibration and orbit-attitude determination of Hayabusa2’s deployable camera (DCAM3)

Hayabusa2’s deployable camera (DCAM3) was deployed from the spacecraft near the asteroid Ryugu to successfully monitor the artificial cratering experiment. Scientific analyses of the observed impact ejecta require accurate determination of the camera’s position and orientation. However, in contrast to conventional spacecraft operations, DCAM3 lacked the capability to acquire orbit tracking data and attitude sensor data due to its limited onboard resources. Even though the optical images obtained by DCAM3 itself are the only source of geometric information, they were subject to significant distortion caused by the rolling shutter effect and lens distortion. This research is, therefore, designed to simultaneously estimate the image distortion and the orbital and attitude motions of DCAM3, relying solely on its image data. The relative geometry between the camera and the asteroid is reconstructed from the geographic information of the observed feature points by incorporating distortion effects. The proposed method involves segmenting the geometry reconstruction, with more than ten thousand feature measurements, into smaller-scale least-squares problems. The image-based estimation yields consistent distortion, orbit, and attitude solutions with pixel-scale accuracy. The analysis results indicate that DCAM3 experienced significant nutation of its optical axis during ballistic flight in a semi-elliptical orbit. In addition, these motions were sensitive enough to simultaneously determine system parameters, such as the gravitational parameter of Ryugu and the inertia moment of DCAM3. This paper illustrates the potential of deployable camera systems as a promising option for enhancing the scientific and engineering aspects of asteroid exploration.

Manifold Peaks Nonnegative Matrix Factorization.

Nonnegative matrix factorization (NMF) has attracted increasing interest for its high interpretability in recent years. It is shown that the NMF is closely related to fuzzy k -means clustering, where the basis matrix represents the cluster centroids. However, most of the existing NMF-based clustering algorithms often have their decomposed centroids deviate away from the data manifold, which potentially undermines the clustering results, especially when the datasets lie on complicated geometric structures. In this article, we present a manifold peaks NMF (MPNMF) for data clustering. The proposed approach has the following advantages: 1) it selects a number of MPs to characterize the backbone of the data manifold; 2) it enforces the centroids to lie on the original data manifold, by restricting each centroid to be a conic combination of a small number of nearby MPs; 3) it generalizes the graph smoothness regularization to guide the local graph construction; and 4) it solves a general problem of quadratic regularized nonnegative least squares (NNLSs) with group l0 -norm constraint and further develops an efficient optimization algorithm to solve the objective function of the MPNMF. Extensive experiments on both synthetic and real-world datasets demonstrate the effectiveness of the proposed approach.

Expedited re-design of multi-band passive microwave circuits using orthogonal scaling directions and gradient-based tuning

Geometry scaling of microwave circuits is an essential but challenging task. In particular, the employment of a given passive structure in a different application area often requires re-adjustment of the operating frequencies/bands while maintaining top performance. Achieving this necessitates the utilization of numerical optimization methods. Nonetheless, if the intended frequencies are distant from the ones at the starting point, local search procedures tend to fail, whereas global search algorithms are computationally expensive. As recently demonstrated, a combination of large-scale concurrent geometry parameter scaling with intermittent local tuning allows for dependable re-design of high-frequency circuits at low CPU costs. Unfortunately, the procedure is only applicable to single-band structures due to synchronized modifications of all operating bands under scaling. This article discusses a novel procedure that leverages a similar overall concept, but allows for independent control of all center frequencies. To achieve this goal, an automated decision-making procedure is developed in which a set of orthogonal scaling directions are determined based on their effect on individual circuit bands, and using auxiliary optimization sub-problems. The scaling range is then automatically computed by solving an appropriately-defined least-square design relocation problem. The methodology introduced in the work is illustrated using two planar passive devices. In both cases, wide-range operating frequency re-design has been demonstrated and favorably compared to conventional gradient-based tuning. Furthermore, the presented procedure has been shown to be computationally efficient. It is also easy to implement and integrate with a variety of gradient-based optimization procedures of a descent type.

Derivative-free bound-constrained optimization for solving structured problems with surrogate models

We propose and analyze a model-based derivative-free (DFO) algorithm for solving bound-constrained optimization problems where the objective function is the composition of a smooth function and a vector of black-box functions. We assume that the black-box functions are smooth and the evaluation of them is the computational bottleneck of the algorithm. The distinguishing feature of our algorithm is the use of approximate function values at interpolation points which can be obtained by an application-specific surrogate model that is cheap to evaluate. As an example, we consider the situation in which a sequence of related optimization problems is solved and present a regression-based approximation scheme that uses function values that were evaluated when solving prior problem instances. In addition, we propose and analyze a new algorithm for obtaining interpolation points that handles unrelaxable bound constraints. Our numerical results show that our algorithm outperforms a state-of-the-art DFO algorithm for solving a least-squares problem from a chemical engineering application when a history of black-box function evaluations is available.