Problem Of Fitting Data Research Articles

Improving the Generalization Abilities of Constructed Neural Networks with the Addition of Local Optimization Techniques

Constructed neural networks with the assistance of grammatical evolution have been widely used in a series of classification and data-fitting problems recently. Application areas of this innovative machine learning technique include solving differential equations, autism screening, and measuring motor function in Parkinson’s disease. Although this technique has given excellent results, in many cases, it is trapped in local minimum and cannot perform satisfactorily in many problems. For this purpose, it is considered necessary to find techniques to avoid local minima, and one technique is the periodic application of local minimization techniques that will adjust the parameters of the constructed artificial neural network while maintaining the already existing architecture created by grammatical evolution. The periodic application of local minimization techniques has shown a significant reduction in both classification and data-fitting problems found in the relevant literature.

Full waveform inversion with smoothing of dilated convolutions

Abstract Full waveform inversion (FWI) is a method for building subsurface models with high resolution by optimizing a data-fitting problem. However, it is still a challenge to apply the FWI method to complicated subsurface models, because FWI is a highly ill-fitting inversion problem. When the simulated data differ from the observed data by more than half a cycle, FWI tends to suffer from the cycle-skipping problems and get trapped in a local minimum. To help the inversion to converge to the accurate subsurface model, we typically build a reasonable initial model or use low-frequency data to start our inversion. Here, we developed a novel technique called smoothing of dilated convolutions inversion (SDCI) to generate low-frequency components from seismic data to recover low-wavenumber components of the subsurface. In the theory part, we first present the objective function of the SDCI method and then derive the gradient of the objective function relative to the velocity using the adjoint-state approach. We then apply the method to a set of simulated data from the Marmousi model. The SDCI method can reasonably estimate the subsurface model even if the simulated seismic record does not contain information below 5 Hz. The numerical examples prove the effectiveness and feasibility of the proposed method. From the SDCI results, the FWI method recovers the velocity with higher accuracy and resolution.

On diagonally structured scheme for nonlinear least squares and data-fitting problems

Recently, structured nonlinear least-squares (NLS) based algorithms gained considerable emphasis from researchers; this attention may result from increasingly applicable areas of these algorithms in different science and engineering domains. In this article, we coined a new efficient structured-based NLS algorithm. We developed a diagonal Hessian-based formulation for solving NLS problems. We derived the quasi-Newton update based on a diagonal matrix scheme subject to a modified structured secant condition. Also, we show that the algorithm’s search direction satisfies a sufficient descent condition under some standard assumptions. Subsequently, we also prove the global convergence of the algorithm and then eventually show its linear convergence rate for strongly convex functions. Furthermore, to show case the proposed algorithm’s performance, we experimented numerically by comparing it with other approaches on some benchmark test functions available in the literature. Finally, the introduced scheme is applied to solve some data-fitting problems

Cardinality-constrained structured data-fitting problems

Cardinality-constrained structured data-fitting problems

Time-domain extended-source full-waveform inversion: Algorithm and practical workflow

Extended-source full-waveform inversion (ES-FWI) first computes wavefields with data-driven source extensions such that the simulated data in inaccurate velocity models match the observed counterpart sufficiently well to prevent cycle skipping. Then, the source extensions are minimized to update the model parameters toward the true medium. This two-step workflow is iterated until data and sources are matched. It has been recently indicated that the source extensions are the least-squares solutions of the scattered data-fitting problem. As a result, the source extensions are computed by propagating backward in time the deconvolved data residuals by the damped data-domain Hessian of the scattered data-fitting problem. Estimating these weighted data residuals is the main computational bottleneck of time-domain ES-FWI. To mitigate this burden, we approximate the inverse data-domain Hessian by mono- and multidimensional matching filters with two simulations per source. We implement time-domain ES-FWI with the alternating-direction method of multipliers and total-variation regularization. Moreover, we apply ES-FWI with a multiscale approach involving frequency continuation and layer stripping, with the latter being implemented with an offset-time-dependent weighting operator. In this framework, we further regularize the inversions while mitigating their computational burden by matching the grid interval to the frequency bandwidth. Finally, the overall workflow combines ES-FWI and classical FWI during the early and late stages of the multiscale approach, respectively. We illustrate that the sensitivity of ES-FWI to the accuracy of the approximated inverse data-domain Hessian depends on the complexity of the targeted model, the data anatomy, and the accuracy of the starting model. In the case of the 2004 BP salt model, we determine that the layer stripping is necessary when the inverse data-domain Hessian is approximated by a 2D Gabor matching filter and the starting model is crude, whereas this feature is not necessary with the Marmousi II model.

Target-oriented high-resolution elastic full-waveform inversion with an elastic redatuming method

Characterizing the elastic properties in deep-buried reservoirs beneath complex overburden structures remains challenging for seismic inversion. Elastic full-waveform inversion (FWI) is capable of quantitatively estimating the subsurface elastic properties with a reasonably high resolution. However, elastic FWI using high frequencies is computationally expensive because fine discretization is required to stabilize the wavefield simulation. In addition, it is challenging for elastic FWI to obtain the high-resolution inversion results of the deep targets due to complex overburden structures and limited energy illumination of the target of interest. To overcome these limitations, we propose a target-oriented high-resolution elastic FWI scheme by using the estimated elastic data for a virtual survey deployed only above a zone of interest. Specifically, we have developed an elastic redatuming approach to retrieve virtual elastic data by solving an iterative least-squares data-fitting problem. In addition to the elastic multicomponent data recorded at the acquisition surface, an estimate of the overburden model is required. A synthetic data example indicates that an overburden model estimated by elastic FWI with a low-frequency band enables our elastic redatuming approach to reconstruct the virtual elastic data representing the seismic reflection response from the target zone. We then perform regularized elastic FWI by using the redatumed elastic data to estimate the elastic properties in the target zone with reduced computational cost. In the numerical examples, the Marmousi2 synthetic data and 2D Volve field data are used to demonstrate the feasibility and practicality of the proposed method.

What Kinds of Functions Do Deep Neural Networks Learn? Insights from Variational Spline Theory

We develop a variational framework to understand the properties of functions learned by fitting deep neural networks with rectified linear unit (ReLU) activations to data. We propose a new function space, which is related to classical bounded variation-type spaces, that captures the compositional structure associated with deep neural networks. We derive a representer theorem showing that deep ReLU networks are solutions to regularized data-fitting problems over functions from this space. The function space consists of compositions of functions from the Banach space of second-order bounded variation in the Radon domain. This Banach space has a sparsity-promoting norm, giving insight into the role of sparsity in deep neural networks. The neural network solutions have skip connections and rank-bounded weight matrices, providing new theoretical support for these common architectural choices. The variational problem we study can be recast as a finite-dimensional neural network training problem with regularization schemes related to the notions of weight decay and path-norm regularization. Finally, our analysis builds on techniques from variational spline theory, providing new connections between deep neural networks and splines.

Least-Squares Reverse Time Migration in Imaging Domain Based on Global Space-Varying Deconvolution

The classical least-squares migration (LSM) translates seismic imaging into a data-fitting optimization problem to obtain high-resolution images. However, the classical LSM is highly dependent on the precision of seismic wavelet and velocity models, and thus it suffers from an unstable convergence and excessive computational costs. In this paper, we propose a new LSM method in the imaging domain. It selects a spatial-varying point spread function to approximate the accurate Hessian operator and uses a high-dimensional spatial deconvolution algorithm to replace the common-used iterative inversion. To keep a balance between the inversion precision and the computational efficiency, this method is implemented based on the strategy of regional division, and the point spread function is computed using only one-time demigration/migration and inverted individually in each region. Numerical experiments reveal the differences in the spatial variation of point spread functions and highlight the importance to use a space-varying deconvolution algorithm. A 3D field case in Northwest China can demonstrate the effectiveness of this method on improving spatial resolution and providing better characterizations for small-scale fracture and cave units of carbonate reservoirs.

Secant variable projection method for solving nonnegative separable least squares problems

The variable projection method is a classical and efficient method for solving separable nonlinear least squares (SNLLS) problems. However, it is hard to handle the constrained SNLLS problems since the explicit form of the Jacobian matrix is required in each iteration. In this paper, we propose a secant variable projection (SVP) method, which employs a rank-one update to estimate the Jacobian matrices. The main advantages of our method are efficiency and ease of applicability to constrained SNLLS problems. The local convergence of our SVP method is also analyzed. Finally, some data fitting and image processing problems are solved to compare the performance of our proposed method with the classical variable projection method. Numerical results illustrate the efficiency and stability of our proposed SVP method in solving the SNLLS problems arising from the blind deconvolution problems.

Maximum compatibility method for data fitting under interval uncertainty

For the linear regression model, the data-fitting problem under the interval uncertainty of the data is studied. As an estimate of the linear function parameters, it is proposed to take their values that deliver the maximum for the so-called recognizing functional of the parameter set compatible with the data (the maximum compatibility method). The properties of the recognizing functional, its interpretation, and the properties of the estimates obtained using the maximum compatibility method are investigated. The relationships to other data analysis methods are discussed, and a practical electrochemistry problem is solved.

Full-waveform inversion via source-receiver extension

Full-waveform inversion produces highly resolved images of the subsurface and quantitative estimation of seismic wave velocity, provided that its initial model is kinematically accurate at the longest data wavelengths. If this initialization constraint is not satisfied, iterative model updating tends to stagnate at kinematically incorrect velocity models producing suboptimal images. The source-receiver extension overcomes this “cycle-skip” pathology by modeling each trace with its own proper source wavelet, permitting a good data fit throughout the inversion process. Because source wavelets should be constant (or vary systematically) across a shot gather, a measure of source trace dependence, for example, the mean square of the signature-deconvolved wavelet scaled by time lag, can be minimized to update the velocity model. For kinematically simple data, such measures of wavelet variance are mathematically equivalent to traveltime misfit. Thus, the model obtained by source-receiver extended inversion is close to that produced by traveltime tomography, even though the process uses no picked times. For more complex data, in which energy travels from source to receiver by multiple raypaths, Green’s function spectral notches may lead to slowly decaying trace-dependent wavelets with energy at time lags unrelated to traveltime error. Tikhonov regularization of the data-fitting problem suppresses these large-lag signals. Numerical examples suggest that this regularized formulation of source-receiver extended inversion is capable of recovering reasonably good velocity models from synthetic transmission and reflection data without stagnation at suboptimal models encountered by standard full-waveform inversion, but with essentially the same computational cost.

Derivative-Free Optimization of Rate Parameters of Capsid Assembly Models from Bulk in Vitro Data.

The assembly of virus capsids proceeds by a complicated cascade of association and dissociation steps, the great majority of which cannot be directly experimentally observed. This has made capsid assembly a rich field for computational models, but there are substantial obstacles to model inference for such systems. Here, we describe progress on fitting kinetic rate constants defining capsid assembly models to experimental data, a difficult data-fitting problem because of the high computational cost of simulating assembly trajectories, the stochastic noise inherent to the models, and the limited and noisy data available for fitting. We evaluate the merits of data-fitting methods based on derivative-free optimization (DFO) relative to gradient-based methods used in prior work. We further explore the advantages of alternative data sources through simulation of a model of time-resolved mass spectrometry data, a technology for monitoring bulk capsid assembly that can be expected to provide much richer data than previously used static light scattering approaches. The results show that advances in both the data and the algorithms can improve model inference. More informative data sources lead to high-quality fits for all methods, but DFO methods show substantial advantages on less informative data sources that better represent current experimental practice.

Wavelet-Based Weighted LASSO and Screening Approaches in Functional Linear Regression

One useful approach for fitting linear models with scalar outcomes and functional predictors involves transforming the functional data to wavelet domain and converting the data-fitting problem to a variable selection problem. Applying the LASSO procedure in this situation has been shown to be efficient and powerful. In this article, we explore two potential directions for improvements to this method: techniques for prescreening and methods for weighting the LASSO-type penalty. We consider several strategies for each of these directions which have never been investigated, either numerically or theoretically, in a functional linear regression context. We compare the finite-sample performance of the proposed methods through both simulations and real-data applications with both 1D signals and 2D image predictors. We also discuss asymptotic aspects. We show that applying these procedures can lead to improved estimation and prediction as well as better stability. Supplementary materials for this article are available online.

A Fast L<sub>1</sub> Linear Estimator and Its Application on Predictive Deconvolution

The L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> linear estimator which is used to solve the linear problem by minimizing the L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> norm in the data-fitting term does better than traditional least square (LS) methods in applications where the residual vector is super-Gaussian or contains outliers. We propose a fast L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> linear estimation algorithm by first translating the L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> norm data-fitting problem into a L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> norm regularized L <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sub> norm data-fitting problem and then solving the equivalent problem by fast iterative shrinkage-thresholding algorithm (FISTA). The method of the equivalence is carefully chosen and designed to achieve sufficiently low computational complexity of each FISTA iteration. The commonly used iterative reweighted least square (IRLS) algorithm is used as a benchmark in this letter. In comparison with IRLS, our numerical experiments show that the proposed algorithm is 5-11 times faster when achieving the same estimation accuracy. To demonstrate the performance of the proposed algorithm, we apply it on seismic predictive deconvolution. Both synthetic and real field data examples show that our method outperforms the IRLS-based and the traditional LS-based seismic predictive deconvolution method.

Applying Derivative-Free Optimization to Fit Kinetic Parameters of Viral Capsid Self-Assembly Models from Multi-Source Bulk in vitro Data

Virus capsid assembly has been a powerful model system for biological self-assembly in general due to the combination of experimental tractability but complicated pathway space. Detailed experimental resolution of viral assembly processes, however, has so far proven impossible. Computational approaches have provided a solution, allowing us to learn models of assembly consistent with indirect experimental measures of bulk in vitro assembly and thus fill the gaps between coarse-grained experimental measurements and detailed theoretical models. Nonetheless, accurate simulation predictions rely on building accurate models, which has proven to be a challenging data-fitting problem due to the high computational cost of simulating capsid assembly trajectories, high stochastic noise inherent to the system, and limited and generally noisy experimental data available. Here, we describe progress in learning accurate kinetic models of capsid assembly systems by computationally fitting assembly simulations to experimental data. We previously developed a heuristic optimization approach to learn rate parameters of coat-coat interactions by minimizing the deviation between real and simulated static light scattering measurements. We now show that one can substantially improve fitting to light scattering data using an alternative class of methods called derivative-free optimization, designed to deal with challenges of costly, noisy computations. Simultaneously, simulated exploration of potential alternative sources of experimental data for monitoring bulk assembly (e.g, non-covalent mass spectrometry) suggest that other feasible technologies providing richer data on assembly progress can more precisely pin down true parameters and assembly pathways. Advancing such simulation-based data fitting methods provides a general technology for greatly enhancing our ability to learn fine-scale details of complex assembly processes from experimental data, a strategy with potential application to developing accurate quantitative models of numerous other assembly systems found through biology.

STOCHASTIC GRADIENT METHODS FOR UNCONSTRAINED OPTIMIZATION

This papers presents an overview of gradient based methods for minimization of noisy functions. It is assumed that the objective functions is either given with error terms of stochastic nature or given as the mathematical expectation. Such problems arise in the context of simulation based optimization. The focus of this presentation is on the gradient based Stochastic Approximation and Sample Average Approximation methods. The concept of stochastic gradient approximation of the true gradient can be successfully extended to deterministic problems. Methods of this kind are presented for the data fitting and machine learning problems.

Fitting data using optimal Hermite type cubic interpolating splines

In this work we obtain a new optimal property for cubic interpolating splines of Hermite type applied to data-fitting problems. The existence and uniqueness of the Hermite type cubic spline with minimal quadratic oscillation in average are proved.

Adaptive constraint reduction for convex quadratic programming

We propose an adaptive, constraint-reduced, primal-dual interior-point algorithm for convex quadratic programming with many more inequality constraints than variables. We reduce the computational effort by assembling, instead of the exact normal-equation matrix, an approximate matrix from a well chosen index set which includes indices of constraints that seem to be most critical. Starting with a large portion of the constraints, our proposed scheme excludes more unnecessary constraints at later iterations. We provide proofs for the global convergence and the quadratic local convergence rate of an affine-scaling variant. Numerical experiments on random problems, on a data-fitting problem, and on a problem in array pattern synthesis show the effectiveness of the constraint reduction in decreasing the time per iteration without significantly affecting the number of iterations. We note that a similar constraint-reduction approach can be applied to algorithms of Mehrotra's predictor-corrector type, although no convergence theory is supplied.

Data assimilation in weather forecasting: a case study in PDE-constrained optimization

Variational data assimilation is used at major weather prediction centers to produce the initial conditions for 7- to 10-day weather forecasts. This technique requires the solution of a very large data-fitting problem in which the major element is a set of partial differential equations that models the evolution of the atmosphere over a time window for which observational data has been gathered. Real-time solution of this difficult computational problem requires sophisticated models of atmospheric physics and dynamics, effective use of supercomputers, and specialized algorithms for optimization and linear algebra. The optimization algorithm can be accelerated by using a spectral preconditioner based on the Lanczos method. This paper shows how practical demands of the application dictate the various algorithmic choices that are made in the nonlinear optimization solver, with particular reference to the system in operation at the European Centre for Medium-Range Weather Forecasts.

Model parameter tuning by cross validation and global optimization: application to the wing weight fitting problem

Model parameter tuning is a fundamental step in any data-fitting problem and of great importance in the final quality of the resulting approximation. Two different sets of model parameters will lead to two different interpolation models that behave very differently between the data points even if both sets of parameters lead to perfect interpolation. The main goal of this paper is to discuss the importance of finding the optimal parameters that will lead to the best prediction model of the given data. This task can be hard, particularly when the number of model parameters is high (usually when the dimension of the problem is high). The wing weight fitting problem is used to illustrate the difficulties in obtaining the best possible approximation in practice.