Separable Nonlinear Least-squares Research Articles

An IDE-based nonlinear grey Bernoulli model and applications to daily traffic flow pattern identification

The nonlinear grey Bernoulli model is a powerful tool for modelling and forecasting time series exhibiting a rough inverted U-shape behaviour. Traditionally, it is considered an indirect dynamical model, informed by empirical knowledge that the accumulation of the inverted U-shape series leads to a growth behaviour, which is subsequently characterized by the Bernoulli ordinary differential equation (ODE). However, its mechanism remains incomplete, particularly concerning the modelling paradigm and parameter estimation. To this end, we propose an equivalent and simplified variant in the form of an integro-differential equation (IDE), enabling direct modelling of the inverted U-shaped series and thereby enhancing the interpretability. By taking the IDE as a state equation in the state space framework, separable nonlinear least squares (SLS) and nonlinear least squares (NLS) are respectively formulated to simultaneously estimate structural parameters and the initial condition. Then, by combining the state equation (ODE/IDE) and parameter estimation approach (SLS/NLS), the generated ODE-based indirect paradigms are comprehensively compared with the IDE-based direct ones. Numerical simulations show that (i) IDE-based models outperform ODE-based ones and (ii) NLS outperforms SLS. Real-world applications indicate that the nonlinear grey Bernoulli model combining IDE and NLS can accurately discover the underlying dynamics of daily traffic flow, outperforming the competitive models.

Simultaneous inversion for source field and mantle electrical conductivity using the variable projection approach

Time-varying electromagnetic field observed on the ground or at a spacecraft consists of contributions from (i) electric source currents, such as those in the ionosphere and magnetosphere, and (ii) corresponding fields induced by source currents within the conductive Earth’s interior by virtue of electromagnetic induction. Knowledge about the spatio-temporal structure of inducing currents is a key component in ionospheric and magnetospheric studies, and is also needed in space weather hazard evaluation, whereas the induced currents depend on the Earth’s subsurface electrical conductivity distribution and allow us to probe this physical property. In this study, we present an approach that reconstructs the inducing source and subsurface conductivity structures simultaneously, preserving consistency between the two models by exploiting the inherent physical link. To achieve this, we formulate the underlying inverse problem as a separable nonlinear least-squares (SNLS) problem, where inducing current and subsurface conductivity parameters enter as linear and nonlinear model unknowns, respectively. We solve the SNLS problem using the variable projection method and compare it with other conventional approaches. We study the properties of the method and demonstrate its feasibility by simultaneously reconstructing the ionospheric and magnetospheric currents along with a 1-D average mantle conductivity distribution from the ground magnetic observatory data.Graphical

A variable projection method for the general radial basis function neural network

The variable projection (VP) method is a classical and effective method for the separable nonlinear least squares (SNLLS) problem. Training a radial basis function neural network (RBFNN) with only one output neuron by minimizing the sum of the squared errors (SSE) is an SNLLS problem, so that the classical VP method has been applied to RBFNN. However, the one-output-RBFNN (ORBFNN) is just one type of RBFNN, so that the paper proposes a new VP method for the general radial basis function neural network (GRBFNN) which has no limit of the number of the output neurons. The new VP method translates the problem corresponding to minimizing the SSE of GRBFNN into a lower-dimensional optimization problem. We prove theoretically that the set of stationary points of the objective function of the lower-dimensional problem is equivalent to that of the original objective function. In addition, the lower dimension leads to less guesses about the initial point for the new problem. The numerical experiments indicate that, with the same algorithm, minimizing the new objective function converges in fewer iterations and makes both a smaller training error and a testing error than minimizing the original objective function.

A Note on the Connection Between Trek Rules and Separable Nonlinear Least Squares in Linear Structural Equation Models

We show that separable nonlinear least squares (SNLLS) estimation is applicable to all linear structural equation models (SEMs) that can be specified in RAM notation. SNLLS is an estimation technique that has successfully been applied to a wide range of models, for example neural networks and dynamic systems, often leading to improvements in convergence and computation time. It is applicable to models of a special form, where a subset of parameters enters the objective linearly. Recently, Kreiberg et al. (Struct Equ Model Multidiscip J 28(5):725–739, 2021. https://doi.org/10.1080/10705511.2020.1835484) have shown that this is also the case for factor analysis models. We generalize this result to all linear SEMs. To that end, we show that undirected effects (variances and covariances) and mean parameters enter the objective linearly, and therefore, in the least squares estimation of structural equation models, only the directed effects have to be obtained iteratively. For model classes without unknown directed effects, SNLLS can be used to analytically compute least squares estimates. To provide deeper insight into the nature of this result, we employ trek rules that link graphical representations of structural equation models to their covariance parametrization. We further give an efficient expression for the gradient, which is crucial to make a fast implementation possible. Results from our simulation indicate that SNLLS leads to improved convergence rates and a reduced number of iterations.

An Iterative Implementation of Variable Projection for Separable Nonlinear Optimization Problems

The separable nonlinear least-squares (SNLLS) problems considered in this article frequently appear in a wide range of research fields, such as machine learning, computer vision, system identification, and signal processing. The variable projection algorithm proposed by Golub and Pereyra, which reduces the dimension of the parameters by projecting the linear parameters out of the problem, is quite valuable in solving SNLLS problems. Previous implementations of the variable projection algorithm are based on matrix factorization. In this article, we propose an iterative implementation of the variable projection algorithm. Compared with previous implementations based on matrix decomposition, the proposed method can effectively avoid suffering from large condition number of the matrix or even matrix decomposition failure when dealing with ill-posed SNLLS problems. Numerical experiments on real-world data and synthetic data show the efficiency and robustness of the proposed iterative variable projection algorithm.

A Faster Procedure for Estimating SEMs Applying Minimum Distance Estimators With a Fixed Weight Matrix

This study presents a separable nonlinear least squares (SNLLS) implementation of the minimum distance (MD) estimator employing a fixed-weight matrix for estimating structural equation models (SEMs). In contrast to the standard implementation of the MD estimator, in which the complete set of parameters is estimated using nonlinear optimization, the SNLLS implementation allows a subset of parameters to be estimated using (linear) least squares (LS). The SNLLS implementation possesses a number of benefits, such as faster convergence, better performance in ill-conditioned estimation problems, and fewer required starting values. The present work demonstrates that SNLLS, when applied to SEM estimation problems, significantly reduces the estimation time. Reduced estimation time makes SNLLS particularly useful in applications involving some form of resampling, such as simulation and bootstrapping.

Full-waveform inversion with source and receiver coupling effects correction

Coupling factors of sources and receivers vary dramatically due to the strong heterogeneity of the near surface, and they are as important as the model parameters for the inversion success. We have adopted a full-waveform inversion (FWI) scheme that corrects for variable coupling factors while updating the model parameters. A linear inversion is embedded into the scheme to estimate the source and receiver factors and compute the amplitude weights according to the acquisition geometry. After the weights are introduced in the objective function, the inversion falls into the category of separable nonlinear least-squares problems. Hence, we could use the variable projection technique widely used in source estimation problems to invert the model parameter without knowledge of the source and receiver factors. The efficacy of the inversion scheme is demonstrated with two synthetic examples and one real data test.

Insights Into Algorithms for Separable Nonlinear Least Squares Problems.

Separable nonlinear least squares (SNLLS) problems have attracted interest in a wide range of research fields such as machine learning, computer vision, and signal processing. During the past few decades, several algorithms, including the joint optimization algorithm, alternated least squares (ALS) algorithm, embedded point iterations (EPI) algorithm, and variable projection (VP) algorithms, have been employed for solving SNLLS problems in the literature. The VP approach has been proven to be quite valuable for SNLLS problems and the EPI method has been successful in solving many computer vision tasks. However, no clear explanations about the intrinsic relationships of these algorithms have been provided in the literature. In this paper, we give some insights into these algorithms for SNLLS problems. We derive the relationships among different forms of the VP algorithms, EPI algorithm and ALS algorithm. In addition, the convergence and robustness of some algorithms are investigated. Moreover, the analysis of the VP algorithm generates a negative answer to Kaufman's conjecture. Numerical experiments on the image restoration task, fitting the time series data using the radial basis function network based autoregressive (RBF-AR) model, and bundle adjustment are given to compare the performance of different algorithms.

A Method for Solving LiDAR Waveform Decomposition Parameters Based on a Variable Projection Algorithm

Light detection and ranging (LiDAR) is commonly used to create high-resolution maps; however, the efficiency and convergence of parameter estimation are difficult. To address this issue, we evaluated the structural characteristics of received LiDAR signals by decomposing them into Gaussian functions and applied the variable projection algorithm of the separable nonlinear least-squares problem to the process of waveform fitting. First, using a variable projection algorithm, we separated the linear (amplitude) and nonlinear (center position and width) parameters in the Gaussian function model; the linear parameters are expressed with nonlinear parameters by the function. Thereafter, the optimal estimation of the characteristic parameters of the Gaussian function components was transformed into a least-squares problem only comprising nonlinear parameters. Finally, the Levenberg–Marquardt algorithm was used to solve these nonlinear parameters, whereas the linear parameters were calculated simultaneously in each iteration, and the estimation results satisfying the nonlinear least-square criterion were obtained. Five groups of waveform decomposition simulation data and ICESat/GLAS satellite LiDAR waveform data were used for the parameter estimation experiments. During the experiments, for the same accuracy, the separable nonlinear least-squares optimization method required fewer iterations and lesser calculation time than the traditional method of not separating parameters; the maximum number of iterations was reached before the traditional method converged to the optimal estimate. The method of separating variables only required 14 iterations to obtain the optimal estimate, reducing the computational time from 1128 s to 130 s. Therefore, the application of the separable nonlinear least-squares problem can improve the calculation efficiency and convergence speed of the parameter solution process. It can also provide a new method for parameter estimation in the Gaussian model for LiDAR waveform decomposition.

A noise-tolerant model parameterization method for lithium-ion battery management system

A well-parameterized battery model is prerequisite of the model-based estimation and control of lithium-ion battery (LIB). However, the unexpected yet inevitable noises may markedly discount the identification of model parameters in real applications. This paper focuses on the noise-immune and unbiased model parameter identification for LIB. The signal-disturbance interface in LIB model identification is firstly analyzed by reformulating an overdetermined nonlinear system, on the premise of a cautiously-designed instrumental vector estimator. The multi-variable identification is then solved in the framework of a separable nonlinear least squares (SNLS) problem via a novel two-step method combining least squares (LS) and variable projection algorithm (VPA), to co-estimate the noise variances and unbiased model parameters. A numerical solver is further exploited for the proposed LSVPA, giving rise to a recursive and computational efficient algorithmic architecture which is favorable for online applications. The proposed method is validated with both simulations and experiments in terms of the noise tolerance and the parameterization accuracy.

Separable Nonlinear Least-Squares Parameter Estimation for Complex Dynamic Systems.

Nonlinear dynamic models are widely used for characterizing processes that govern complex biological pathway systems. Over the past decade, validation and further development of these models became possible due to data collected via high-throughput experiments using methods from molecular biology. While these data are very beneficial, they are typically incomplete and noisy, which renders the inference of parameter values for complex dynamic models challenging. Fortunately, many biological systems have embedded linear mathematical features, which may be exploited, thereby improving fits and leading to better convergence of optimization algorithms. In this paper, we explore options of inference for dynamic models using a novel method of separable nonlinear least-squares optimization and compare its performance to the traditional nonlinear least-squares method. The numerical results from extensive simulations suggest that the proposed approach is at least as accurate as the traditional nonlinear least-squares, but usually superior, while also enjoying a substantial reduction in computational time.

Regularization methods for separable nonlinear models

Separable nonlinear models frequently arise in system identification, signal analysis, electrical engineering, and machine learning. Their parameter optimization belongs to a class of separable nonlinear least squares (SNLLS) problem. Applying the classical variable projection algorithm to the SNLLS problems may give poor generalization. In order to handle complexity control and ill-conditioned nonlinear least squares problems, we consider in this paper two $$L_2$$ regularization algorithms for the SNLLS problems. The first approach is to directly add a Tikhonov penalty to the objective function of the SNLLS problem. The second approach is to replace the ordinary linear least squares problem in the SNLLS problem by a Tikhonov one. We give their difference from the perspective of Bayesian. Numerical experiments are also presented to compare the performance of the two regularized algorithms. Results show that the first regularization method is more robust than the second one.

Modified Gram-Schmidt Method-Based Variable Projection Algorithm for Separable Nonlinear Models.

Separable nonlinear models are very common in various research fields, such as machine learning and system identification. The variable projection (VP) approach is efficient for the optimization of such models. In this paper, we study various VP algorithms based on different matrix decompositions. Compared with the previous method, we use the analytical expression of the Jacobian matrix instead of finite differences. This improves the efficiency of the VP algorithms. In particular, based on the modified Gram-Schmidt (MGS) method, a more robust implementation of the VP algorithm is introduced for separable nonlinear least-squares problems. In numerical experiments, we compare the performance of five different implementations of the VP algorithm. Numerical results show the efficiency and robustness of the proposed MGS method-based VP algorithm.

A Regularized Variable Projection Algorithm for Separable Nonlinear Least Squares Problems

Separable nonlinear least-squares (SNLLS) problems arise frequently in many research fields, such as system identification and machine learning. The variable projection (VP) method is a very powerful tool for solving such problems. In this paper, we consider the regularization of ill-conditioned SNLLS problems based on the VP method. Selecting an appropriate regularization parameter is difficult because of the nonlinear optimization procedure. We propose to determine the regularization parameter using the weighted generalized cross-validation method at every iteration. This makes the original objective function changing during the optimization procedure. To circumvent this problem, we use an inequation to produce a consistent demand of decreasing at successive iterations. The approximation of the Jacobian of the regularized problem is also discussed. The proposed regularized VP algorithm is tested by the parameter estimation problem of several statistical models. Numerical results demonstrate the effectiveness of the proposed algorithm.

Resolution enhancement of seismic data using spectral modeling based on dominant Ricker components and separable nonlinear least squares

Seismic resolution enhancement has a key role in seismic interpretation, especially in reservoir characterization and geologic interpretation. Exploration seismology has a limitation in temporal resolution and thin-layer detection, so any effort to improve the resolution of available data is valuable. Seismic deconvolution is one of the main steps in seismic processing and is intensively used for enhancing the vertical resolution. In this procedure, the seismic wavelet is compressed in order to decrease wavelet interferences and improve temporal resolution. In the common method of seismic deconvolution — i.e., Wiener deconvolution — the seismic wavelet is estimated by using the early part of autocorrelation of traces. Assuming the reflection series is a random function, the smoothed version of the amplitude spectrum of traces can be considered as the amplitude spectrum of the seismic wavelet. The wavelet spectrum can be properly reconstructed by the linear combination of a number of Ricker wavelets with different peak frequencies. The authors aim to introduce a novel method for resolution improvement based on fitting and finding main Ricker components of the source-wavelet spectrum. This attempt is classified under the group of the separable nonlinear least squares (SNLS), the objective function of which is a combination of linear and nonlinear functions. A variable projection method is applied to recognize the optimum peak frequency of Ricker wavelets involved in the wavelet spectrum. Our synthetic tests indicate that the SNLS algorithm is able to achieve a very close approximation of dominant Ricker components of the source spectrum. Considering the minimum phase assumption of the wavelet, the phase spectrum is also computed by Hilbert transform of the estimated amplitude spectrum. Once the amplitude and phase information are available, the deconvolution operator can be designed. Our synthetic and real tests indicate that, in comparison with Wiener deconvolution, spectral modeling based on dominant Ricker components (SMDRC) has improved the deconvolution outputs.

Data-driven uncertainty quantification of structural systems via B-spline expansion

A data-driven uncertainty quantification scheme relying on B-spline functions is investigated in this study. Instead of predefining basis functions according to the statistics of the uncertain input, as in conventional polynomial chaos expansion (PCE)-based implementations, the method introduced herein takes advantage of the increased flexibility of B-splines materializing a basis function parametrization, which is adaptable to a given input–output data set. Parameter estimation is effectively dealt through a Separable Non-linear Least Squares (SNLS) procedure that allows for simultaneous estimation of both the B-splines’ free knots and the corresponding coefficients of projection. The induced constrained non-quadratic optimization problem is treated through the implementation of a hybrid optimization algorithm that interrelates the advantages of evolutionary and deterministic schemes, exhibiting accelerated convergence and robustness in the search for global optimum. The efficiency of the method is examined through both numerical and experimental case studies, while comparisons with the standard PCE, which relies on the Wiener–Askey scheme, as well as alternative data-driven methods, such as the arbitrary PC, indicate superior performance.

Least Squares Time-Series Synchronization in Image Acquisition Systems.

We consider an acquisition system constituted by an array of sensors scanning an image. Each sensor produces a sequence of readouts, called a time series. In this framework, we discuss the image estimation problem when the time series are affected by noise and by a time shift. In particular, we introduce an appropriate data model and consider the least squares (LS) estimate, showing that it has no closed form. However, the LS problem has a structure that can be exploited to simplify the solution. In particular, based on two known techniques, namely, separable nonlinear LS and alternating LS, we propose and analyze several practical estimation methods. As an additional contribution, we discuss the application of these methods to the data of the photodetector array camera and spectrometer, which is an infrared photometer onboard the Herschel satellite. In this context, we investigate the accuracy and the computational complexity of the methods, using both true and simulated data.

Robust identification of continuous-time models with arbitrary time-delay from irregularly sampled data

This paper presents a new approach to identify continuous-time systems with arbitrary time-delay from irregularly sampled input–output data. It is based on the separable nonlinear least-squares method which combines in a bootstrap manner the iterative optimal instrumental variable method for transfer function model estimation with an adaptive gradient-based technique that searches for the optimal time-delay. Since the objective function may have several local minima with respect to the unknown parameters (especially the time-delay), the initialization requires special attention. Here, a low-pass filtering strategy is used to widen the convergence region around the global minimum. Simulation results are included to show the performance of the proposed method.

Theoretical limits of component identification in a separable nonlinear least-squares problem

We provide theoretical insights into component identification in a separable nonlinear least-squares problem in which the model is a linear combination of nonlinear functions (called components in this paper). Within this research, we assume that the number of components is unknown. The objective of this paper is to understand the limits of component discovery under the assumed model. We focus on two aspects. One is sensitivity analysis referring to the ability of separating regression components from noise. The second is resolution analysis referring to the ability of de-mixing components that have similar location parameters. We use a wavelet transformation that allows to zoom in at different levels of details in the observed data. We further apply these theoretical insights to provide a road map on how to detect components in more realistic settings such as a two-dimensional nuclear magnetic resonance experiment for protein structure determination.

Automated spectral recomposition with application in stratigraphic interpretation

Analyzing seismic attributes in the frequency domain is helpful for reservoir characterization. To analyze the reservoir interval of interest in detail, it is important to capture the seismic response at each frequency subset. Spectral recomposition can be used to extract significant components from the seismic spectrum. We propose a separable nonlinear least-squares algorithm for spectral recomposition, which estimates linear and nonlinear parts automatically in separate steps. Our approach is applied to estimate fundamental signal parameters, peak frequencies, and amplitudes, with which the seismic spectrum can be reconstructed. Automated spectral recomposition helps us visualize frequency-dependent geologic features on cross sections and time slices by extracting significant frequency components. Spectral recomposition can also indicate how frequency contents attenuate with time.