Least-squares Projection Research Articles

Meshfree RBF–FD methods for numerical simulation of PDE problems

The meshfree RBF–FD method is becoming an increasingly popular discretization method for challenging PDE problems due to its flexibility with respect to geometry and its ease of implementation. Using a combination of polyharmonic spline basis functions and polynomial basis functions has proven to be particularly successful with a convergence rate depending on the polynomial degree and good behavior at boundaries due to the contribution of the spline part. A challenge that has been observed is that derivative boundary conditions can give rise to large errors. A known remedy for this problem is to introduce one or more layers of ghost points, which improves the boundary approximations. In a recent paper, Tominec, Larsson, Heryudono (2021), it was shown that using oversampling is another effective way to handle this problem. Oversampling also allows for theoretical analysis of the method, which can then be viewed as a discretization of a continuous least-squares projection. By looking deeper into the error estimates we gain an understanding about how to best implement the method.

Robust Reduced Order Thau Observer With the Adaptive Fault Estimator for the Unmanned Air Vehicles

Developing fault detection and diagnoses algorithms for the unmanned air vehicles such as the quadrotors is challenging since they are intrinsically non-linear, time-varying, unstable, and uncertain. This paper develops a reduced order Thau observer by only considering the uncertain rotational dynamics, which are re-constructed as the dominant linear and non-linear for the design purpose. Therefore, the proposed Thau observer is just third order and can reveal a rotational state estimation error in the presence of the quadrotor faults. This paper also equips the proposed Thau observer with a simple online adaptive fault estimation law, which is able to recognize up to two faulty actuators instantly using the estimated rotational state error. Lyapunov analysis confirms the error convergence in both the Thau observer states and the adaptive fault estimates. In addition, this paper constructs a batch type least-squares projection approach to quantify the magnitude percentages of the actuator failures. Moreover, to show the feasibility of the proposed algorithm, this paper extensively analyses the fault detection and diagnosis results performed in the simulation and real-time environments. Finally, to demonstrate the superiority of the proposed algorithm, it is compared with a recent Kalman filter based quadrotor fault estimation research under the equal conditions.

Understanding High Dimensional Spaces through Visual Means Employing Multidimensional Projections

Data visualisation helps understanding data represented by multiple variables, also called features, stored in a large matrix where individuals are stored in lines and variable values in columns. These data structures are frequently called multidimensional spaces. A large set of mathematical tools, named frequently as multidimensional projections, aim to map such large spaces into 'visual spaces', that is, to 2 or 3 dimensions, where the aspect of that space can be visualised. While the final product is intuitive in that proximity between points - or iconic representation of points - indicate similarity relationships in the original space, understanding the formulation of the projection methods many times escapes researchers. In this paper, we illustrate ways of employing the visual results of multidimensional projection algorithms to understand and fine-tune the parameters of their mathematical framework. Some of the common mathematical common to these approaches are Laplacian matrices, Euclidian distance, Cosine distance, and statistical methods such as Kullback-Leibler divergence, employed to fit probability distributions and reduce dimensions. Two of the relevant algorithms in the data visualisation field are t-distributed stochastic neighbourhood embedding (t-SNE) and Least-Square Projection (LSP). These algorithms can be used to understand several ranges of mathematical functions including their impact on datasets. In this article, mathematical parameters of underlying techniques such as Principal Component Analysis (PCA) behind t-SNE and mesh reconstruction methods behind LSP are adjusted to reflect the properties afforded by the mathematical formulation. The results, supported by illustrative methods of the processes of LSP and t-SNE, are meant to inspire students in understanding the mathematics behind such methods, in order to apply them in effective data analysis tasks in multiple applications.

Component-Based Reduced Order Modeling of Large-Scale Complex Systems

Large-scale engineering systems, such as propulsive engines, ship structures, and wind farms, feature complex, multi-scale interactions between multiple physical phenomena. Characterizing the operation and performance of such systems requires detailed computational models. Even with advances in modern computational capabilities, however, high-fidelity (e.g., large eddy) simulations of such a system remain out of reach. In this work, we develop a reduced‐order modeling framework to enable accurate predictions of large-scale systems. We target engineering systems which are difficult to simulate at a high-enough level of fidelity, but are decomposable into different components. These components can be modeled using a combination of strategies, such as reduced-order models (ROM) or reduced-fidelity full-order models (RF-FOM). Component-based training strategies are developed to construct ROMs for each individual component. These ROMs are then integrated to represent the full system. Notably, this approach only requires high-fidelity simulations of a much smaller computational domain. System-level responses are mimicked via external boundary forcing during training. Model reduction is accomplished using model-form preserving least-squares projections with variable transformation (MP-LSVT) (Huang et al., Journal of Computational Physics, 2022, 448: 110742). Predictive capabilities are greatly enhanced by developing adaptive bases which are locally linear in time. The trained ROMs are then coupled and integrated into the framework to model the full large-scale system. We apply the methodology to extremely complex flow physics involving combustion dynamics. With the use of the adaptive basis, the framework is demonstrated to accurately predict local pressure oscillations, time-averaged and RMS fields of target state variables, even with geometric changes.

A Missing Data Reconstruction Method Using an Accelerated Least-Squares Approximation with Randomized SVD

An accelerated least-squares approach is introduced in this work by incorporating a greedy point selection method with randomized singular value decomposition (rSVD) to reduce the computational complexity of missing data reconstruction. The rSVD is used to speed up the computation of a low-dimensional basis that is required for the least-squares projection by employing randomness to generate a small matrix instead of a large matrix from high-dimensional data. A greedy point selection algorithm, based on the discrete empirical interpolation method, is then used to speed up the reconstruction process in the least-squares approximation. The accuracy and computational time reduction of the proposed method are demonstrated through three numerical experiments. The first two experiments consider standard testing images with missing pixels uniformly distributed on them, and the last numerical experiment considers a sequence of many incomplete two-dimensional miscible flow images. The proposed method is shown to accelerate the reconstruction process while maintaining roughly the same order of accuracy when compared to the standard least-squares approach.

Boosted optimal weighted least-squares

This paper is concerned with the approximation of a function u u in a given subspace V m V_m of dimension m m from evaluations of the function at n n suitably chosen points. The aim is to construct an approximation of u u in V m V_m which yields an error close to the best approximation error in V m V_m and using as few evaluations as possible. Classical least-squares regression, which defines a projection in V m V_m from n n random points, usually requires a large n n to guarantee a stable approximation and an error close to the best approximation error. This is a major drawback for applications where u u is expensive to evaluate. One remedy is to use a weighted least-squares projection using n n samples drawn from a properly selected distribution. In this paper, we introduce a boosted weighted least-squares method which allows to ensure almost surely the stability of the weighted least-squares projection with a sample size close to the interpolation regime n = m n=m . It consists in sampling according to a measure associated with the optimization of a stability criterion over a collection of independent n n -samples, and resampling according to this measure until a stability condition is satisfied. A greedy method is then proposed to remove points from the obtained sample. Quasi-optimality properties in expectation are obtained for the weighted least-squares projection, with or without the greedy procedure. The proposed method is validated on numerical examples and compared to state-of-the-art interpolation and weighted least-squares methods.

Model reduction for multi-scale transport problems using model-form preserving least-squares projections with variable transformation

A projection-based formulation is presented for non-linear model reduction of problems with extreme scale disparity. The approach allows for the selection of an arbitrary, but complete, set of solution variables while preserving the structure of the governing equations. Least-squares-based minimization is leveraged to guarantee symmetrization and discrete consistency with the full-order model (FOM). Two levels of scaling are used to achieve the conditioning required to effectively handle problems with extremely disparate physical phenomena, characterized by extreme stiffness in the system of equations. The formulation – referred to as model-form preserving least-squares with variable transformation (MP-LSVT) – provides global stabilization for both implicit and explicit time integration schemes. To achieve computational efficiency, a pivoted QR decomposition is used with oversampling, and adapted to the MP-LSVT method. The framework is demonstrated in representative two- and three-dimensional reacting flow problems, and the MP-LSVT is shown to exhibit improved stability and accuracy over standard projection-based ROM techniques. Physical realizability and local stability are promoted by enforcing limiters in both temperature and species mass fractions. These limiters are demonstrated to be important in eliminating regions of spurious burning, thus enabling the ROMs to provide accurate representations of the heat release rate and flame propagation speed. In the 3D application, it is shown that more than two orders of magnitude acceleration in computational efficiency can be achieved, while also providing reasonable future-state predictions. A key contribution of this work is the development and demonstration of a comprehensive ROM formulation that targets highly challenging multi-scale transport-dominated problems.

Wang and Leng (2016), High-Dimensional Ordinary Least-Squares Projection for Screening Variables, Journal of The Royal Statistical Society Series B, 78, 589–611

Wang and Leng (2016), High-Dimensional Ordinary Least-Squares Projection for Screening Variables, Journal of The Royal Statistical Society Series B, 78, 589–611

Application of chemometric methods to the analysis of multimodal chemical images of biological tissues.

Current histology techniques, such as tissue staining or histochemistry protocols, provide very limited chemical information about the tissues. Chemical imaging technologies such as infrared, Raman, and mass spectrometry imaging, are powerful analytical techniques with a huge potential in describing the chemical composition of sample surfaces. In this work, three images of the same tissue slice using matrix-assisted laser desorption/ionization (MALDI) mass spectrometry, infrared microspectroscopy, and an RGB picture from a conventional hematoxylin/eosin (H/E) staining are simultaneously analyzed. These fused images were analyzed by multivariate curve resolution-alternating least squares (MCR-ALS), which provided, for each component, its distribution within the tissue surface, its IR spectrum fingerprint, its characteristic mass values, and the contribution of the RGB channels of the H/E staining. Compared with the individual analysis of each of the images alone, the fusion of the three images showed the relationship between the different types of chemical/biological information and enabled a better interpretation of the tissue under study. In addition, the least-squares projection of the MCR-ALS resolved spectra of components at low spatial resolution onto the IR and RBG images at high spatial resolution, provided a better delimitation of the sample constituents on the image, giving a more precise description of their distribution on the investigated tissue. The application of this procedure can be of interest in different research areas in which a good description of the spatial distribution of the chemical constituents of the samples is needed, such as in biomedicine, food, or environmental research.

Training Neural Networks as Learning Data-adaptive Kernels: Provable Representation and Approximation Benefits

Consider the problem: given the data pair drawn from a population with , specify a neural network model and run gradient flow on the weights over time until reaching any stationarity. How does ft , the function computed by the neural network at time t, relate to , in terms of approximation and representation? What are the provable benefits of the adaptive representation by neural networks compared to the pre-specified fixed basis representation in the classical nonparametric literature? We answer the above questions via a dynamic reproducing kernel Hilbert space (RKHS) approach indexed by the training process of neural networks. Firstly, we show that when reaching any local stationarity, gradient flow learns an adaptive RKHS representation and performs the global least-squares projection onto the adaptive RKHS, simultaneously. Secondly, we prove that as the RKHS is data-adaptive and task-specific, the residual for lies in a subspace that is potentially much smaller than the orthogonal complement of the RKHS. The result formalizes the representation and approximation benefits of neural networks. Lastly, we show that the neural network function computed by gradient flow converges to the kernel ridgeless regression with an adaptive kernel, in the limit of vanishing regularization. The adaptive kernel viewpoint provides new angles of studying the approximation, representation, generalization, and optimization advantages of neural networks.

A Factorization of Least-Squares Projection Schemes for Ill-Posed Problems

Abstract We give a factorization formula to least-squares projection schemes, from which new convergence conditions together with formulas estimating the rate of convergence can be derived. We prove that the convergence of the method (including the rate of convergence) can be completely determined by the principal angles between T † ⁢ T ⁢ ( X n ) {T^{\dagger}T(X_{n})} and T * ⁢ T ⁢ ( X n ) {T^{*}T(X_{n})} , and the principal angles between X n ∩ ( 𝒩 ⁢ ( T ) ∩ X n ) ⊥ {X_{n}\cap(\mathcal{N}(T)\cap X_{n})^{\perp}} and ( 𝒩 ⁢ ( T ) + X n ) ∩ 𝒩 ⁢ ( T ) ⊥ {(\mathcal{N}(T)+X_{n})\cap\mathcal{N}(T)^{\perp}} . At the end, we consider several specific cases and examples to further illustrate our theorems.

Manifold Approximation by Moving Least-Squares Projection (MMLS)

In order to avoid the curse of dimensionality, frequently encountered in big data analysis, there has been vast development in the field of linear and nonlinear dimension reduction techniques in recent years. These techniques (sometimes referred to as manifold learning) assume that the scattered input data is lying on a lower-dimensional manifold; thus the high dimensionality problem can be overcome by learning the lower dimensionality behavior. However, in real-life applications, data is often very noisy. In this work, we propose a method to approximate $$\mathcal {M}$$ a d-dimensional $$C^{m+1}$$ smooth submanifold of $$\mathbb {R}^n$$ ( $$d \ll n$$ ) based upon noisy scattered data points (i.e., a data cloud). We assume that the data points are located “near” the lower-dimensional manifold and suggest a nonlinear moving least-squares projection on an approximating d-dimensional manifold. Under some mild assumptions, the resulting approximant is shown to be infinitely smooth and of high approximation order (i.e., $$\mathcal {O}(h^{m+1})$$ , where h is the fill distance and m is the degree of the local polynomial approximation). The method presented here assumes no analytic knowledge of the approximated manifold and the approximation algorithm is linear in the large dimension n. Furthermore, the approximating manifold can serve as a framework to perform operations directly on the high-dimensional data in a computationally efficient manner. This way, the preparatory step of dimension reduction, which induces distortions to the data, can be avoided altogether.

Parameter-Refined OMP for Compressive Radar Imaging of Rotating Targets

This paper presents a new parameter-refined orthogonal matching pursuit method to tackle the dictionary mismatch problem arising in compressive radar imaging of a rotating target with off-grid scatterers. The main idea is to replace the linear least-squares projection step of the conventional orthogonal matching pursuit (OMP) method with a nonlinear least-squares procedure which jointly estimates the off-grid positions and reflectivities of true scatterers. Simulations are presented to corroborate the performance advantages of the proposed method.

Evaluation of Multivariate Classification Models for Analyzing NMR Metabolomics Data.

Analytical techniques such as NMR and mass spectrometry can generate large metabolomics data sets containing thousands of spectral features derived from numerous biological observations. Multivariate data analysis is routinely used to uncover the underlying biological information contained within these large metabolomics data sets. This is typically accomplished by classifying the observations into groups (e.g., control versus treated) and by identifying associated discriminating features. There are a variety of classification models to select from, which include some well-established techniques (e.g., principal component analysis [PCA], orthogonal projection to latent structure [OPLS], or partial least-squares projection to latent structures [PLS]) and newly emerging machine learning algorithms (e.g., support vector machines or random forests). However, it is unclear which classification model, if any, is an optimal choice for the analysis of metabolomics data. Herein, we present a comprehensive evaluation of five common classification models routinely employed in the metabolomics field and that are also currently available in our MVAPACK metabolomics software package. Simulated and experimental NMR data sets with various levels of group separation were used to evaluate each model. Model performance was assessed by classification accuracy rate, by the area under a receiver operating characteristic (AUROC) curve, and by the identification of true discriminating features. Our findings suggest that the five classification models perform equally well with robust data sets. Only when the models are stressed with subtle data set differences does OPLS emerge as the best-performing model. OPLS maintained a high-prediction accuracy rate and a large area under the ROC curve while yielding loadings closest to the true loadings with limited group separations.

On derivatives of smooth functions represented in multiwavelet bases

We construct high-order derivative operators for smooth functions represented via discontinuous multiwavelet bases. The need for such operators arises in order to avoid artifacts when computing functionals involving high-order derivatives of solutions of integral equations. Previously high-order derivatives had to be formed by repeated application of a first-derivative operator that, while uniquely defined, has a spectral norm that grows quadratically with polynomial order and, hence, greatly amplifies numerical noise (truncation error) in the multiwavelet computation. The new constructions proceed via least-squares projection onto smooth bases and provide substantially improved numerical properties as well as permitting direct construction of high-order derivatives. We employ either b-splines or bandlimited exponentials as the intermediate smooth basis, with the former maintaining the concept of approximation order while the latter preserves the pure imaginary spectrum of the first-derivative operator and provides more direct control over the bandlimit and accuracy of computation. We demonstrate the properties of these new operators via several numerical tests as well as application to a problem in nuclear physics.

AirInsight: Visual Exploration and Interpretation of Latent Patterns and Anomalies in Air Quality Data

Nowadays, huge volume of air quality data provides unprecedented opportunities for analyzing pollution. However, due to the high complexity, most traditional analytical methods focus on abstracting data, so these techniques discard the original structure and limit the understanding of the results. Visual analysis is a powerful technique for exploring unknown patterns since it retains the details of the original data and gives visual feedback to users. In this paper, we focus on air quality data and propose the AirInsight design, an interactive visual analytic system for recognizing, exploring, and summarizing regular patterns, as well as detecting, classifying, and interpreting abnormal cases. Based on the time-varying and multivariate features of air quality data, a dimension reduction method Composite Least Square Projection (CLSP) is proposed, which allows appreciating and interpreting the data patterns in the context of attributes. On the basis of the observed regular patterns, multiple abnormal cases are further detected, including the multivariate anomalies by the proposed Noise Hierarchical Clustering (NHC) method, abruptly changing timestamps by Time diversity (TD) indicator, and cities with unique patterns by the Geographical Surprise (GS) measure. Moreover, we combine TD and GS to group anomalies based on their underlying spatiotemporal correlations. AirInsight includes multiple coordinated views and rich interactive functions to provide contextual information from different aspects and facilitate a comprehensive understanding. In particular, a pair of glyphs are designed that provide a visual representation of the temporal variation in air quality conditions for a user-selected city. Experiments show that CLSP improves the accuracy of Least Square Projection (LSP) and that NHC has the ability to separate noises. Meanwhile, several case studies and task-based user evaluation demonstrate that our system is effective and practical for exploring and interpreting multivariate spatiotemporal patterns and anomalies in air quality data.

Approximate Maximum-likelihood Identification of Linear Systems from Quantized Measurements

We analyze likelihood-based identification of systems that are linear in the parameters from quantized output data; in particular, we propose a method to find approximate maximum-likelihood and maximum-a-posteriori solutions. The method consists of appropriate least-squares projections of the middle point of the active quantization intervals. We show that this approximation maximizes a variational approximation of the likelihood and we provide an upper bound for the approximation error. In a simulation study, we compare the proposed method with the true maximum-likelihood estimate of a finite impulse response model.

Hyperspectral image-based multi-feature integration for TVB-N measurement in pork

Total volatile basic nitrogen (TVB-N) content is an important index used to evaluate the freshness of pork. In this paper, a strategy for measurement of TVB-N content in pork through hyperspectral imaging (HSI) (400–1000 nm) was developed. Firstly, image textural features based on Gabor filter and spectral features were obtained from the hyperspectral image after determining the region of interest. Then, nine feature wavelengths were selected using partial least-squares projection algorithm. And, major components were obtained from the 2D principal component analysis (2DPCA). Finally, a calibration model was established based on major components using least-squares support vector machine to predict TVB-N values. The results of two methods for data fusion, which are 2DPCA and principal component analysis (PCA), are compared. The correlation coefficients of prediction (RP) and root-mean-square errors of prediction (RMSEP) obtained through 2DPCA were 0.955 and 1.86 mg/100 g respectively, which was superior to the results based on PCA (RP = 0.944, RMSEP = 2.07 mg/100 g). Compared to PCA, the residual prediction deviations (RPD) based on 2DPCA was raised from 3.01 to 3.35. Results demonstrated that the proposed model based on 2DPCA exhibited potential for nondestructive detection of TVB-N content in pork.

Performance Analysis of Source Image Estimators in Blind Source Separation

Blind methods often separate or identify signals or signal subspaces up to an unknown scaling factor. Sometimes it is necessary to cope with the scaling ambiguity, which can be done through reconstructing signals as they are received by sensors, because scales of the sensor responses (images) have known physical interpretations. In this paper, we analyze two approaches that are widely used for computing the sensor responses, especially, in Frequency-Domain Independent Component Analysis. One approach is the least-squares projection, while the other one assumes a regular mixing matrix and computes its inverse. Both estimators are invariant to the unknown scaling. Although frequently used, their differences were not studied yet. A goal of this work is to fill this gap. The estimators are compared through a theoretical study, perturbation analysis and simulations. We point to the fact that the estimators are equivalent when the separated signal subspaces are orthogonal, and vice versa. Two applications are shown, one of which demonstrates a case where the estimators yield substantially different results.

Double Least-Squares Projections Method for Signal Estimation

A real-world signal is always corrupted with noise. The separation between a signal and noise is an indispensable step in a variety of signal-analysis applications across different scientific domains. In this paper, we propose a double least-squares projections (DLSPs) method to estimate a signal from the noisy data. The first least-squares projection is to find a signal-dimensional optimal approximation of the noisy data in the least-squares sense. In this step, a rough estimation of the signal is obtained. The second least-squares projection is to find an approximation of a signal in another crossed signal-dimensional space in the least-squares sense. In this step, a much improved signal estimation that is close to orthogonal to the separated noise subspace can be obtained. The DLSP implements projection operation twice to obtain an almost perfect estimation of a signal. The application of the DLSP method in seismic random noise attenuation and signal reconstruction demonstrates the successful performance in seismic data processing.