Smallest Singular Value Research Articles

An Improved Tikhonov-Regularized Variable Projection Algorithm for Separable Nonlinear Least Squares

In this work, we investigate the ill-conditioned problem of a separable, nonlinear least squares model by using the variable projection method. Based on the truncated singular value decomposition method and the Tikhonov regularization method, we propose an improved Tikhonov regularization method, which neither discards small singular values, nor treats all singular value corrections. By fitting the Mackey–Glass time series in an exponential model, we compare the three regularization methods, and the numerically simulated results indicate that the improved regularization method is more effective at reducing the mean square error of the solution and increasing the accuracy of unknowns.

Neuro‐Fuzzy Kinematic Finite‐Fault Inversion: 1. Methodology

AbstractKinematic finite‐fault source inversions aim at resolving the spatio‐temporal evolution of slip on a fault given ground motion recorded on the Earth's surface. This type of inverse problem is inherently ill posed due to two main factors. First, the number of model parameters is typically greater than the number of independent observed data. Second, small singular values are generated by the discretization of the physical rupture process and amplify the effect of noise in the inversion. As a result, one can find different slip distributions that fit the data equally well. This ill posedness can be mitigated by decreasing the number of model parameters, hence improving their relationship to the observed data. In this study, we propose a fuzzy function approximation approach to describe the spatial slip function. In particular, we use an Adaptive Network‐based Fuzzy Inference System (ANFIS) to find the most adequate discretization for the spatial variation of slip on the fault. The fuzzy basis functions and their respective amplitudes are optimized through hybrid learning. We solve this earthquake source problem in the frequency domain, searching for optimal spatial slip distribution independently for each frequency. The approximated frequency‐dependent spatial slip functions are then used to compute the forward relationship between slip on the fault and ground motion. The method is constrained through Tikhonov regularization, requiring a smooth spatial slip variation. We discuss how the number of model parameters can be decreased, while keeping the inversion stable and achieving an adequate resolution. The proposed inversion method is tested using the SIV1‐benchmark exercise.

An unconventional robust integrator for dynamical low-rank approximation

We propose and analyse a numerical integrator that computes a low-rank approximation to large time-dependent matrices that are either given explicitly via their increments or are the unknown solution to a matrix differential equation. Furthermore, the integrator is extended to the approximation of time-dependent tensors by Tucker tensors of fixed multilinear rank. The proposed low-rank integrator is different from the known projector-splitting integrator for dynamical low-rank approximation, but it retains the important robustness to small singular values that has so far been known only for the projector-splitting integrator. The new integrator also offers some potential advantages over the projector-splitting integrator: It avoids the backward time integration substep of the projector-splitting integrator, which is a potentially unstable substep for dissipative problems. It offers more parallelism, and it preserves symmetry or anti-symmetry of the matrix or tensor when the differential equation does. Numerical experiments illustrate the behaviour of the proposed integrator.

On the smallest singular value in the class of unit lower triangular matrices with entries in [−a, a

Abstract Given a real number a ≥ 1, let Kn (a) be the set of all n × n unit lower triangular matrices with each element in the interval [−a, a]. Denoting by λn (·) the smallest eigenvalue of a given matrix, let cn (a) = min {λ n (YYT ) : Y ∈ Kn (a)}. Then c n ( a ) \sqrt {{c_n}\left( a \right)} is the smallest singular value in Kn (a). We find all minimizing matrices. Moreover, we study the asymptotic behavior of cn (a) as n → ∞. Finally, replacing [−a, a] with [a, b], a ≤ 0 < b, we present an open question: Can our results be generalized in this extension?

Random Matrices Generating Large Growth in LU Factorization with Pivoting

We identify a class of random, dense, $n\times n$ matrices for which LU factorization with any form of pivoting produces a growth factor typically of size at least $n/(4 \log n)$ for large $n$. The condition number of the matrices can be arbitrarily chosen, and large growth also happens for the transpose. Previously, no matrices with all these properties were known. The matrices can be generated by the MATLAB function gallery('randsvd',...), and they are formed as the product of two random orthogonal matrices from the Haar distribution with a diagonal matrix having only one diagonal entry different from 1, which lies between 0 and 1 (the “one small singular value” case). Our explanation for the large growth uses the fact that the maximum absolute value of any element of a Haar distributed orthogonal matrix tends to be relatively small for large $n$. We verify the behavior numerically and find that for partial pivoting the actual growth is significantly larger than the lower bound and much larger than the growth observed for random matrices with elements from the uniform [0,1] or standard normal distributions. We show more generally that a rank-1 perturbation to an orthogonal matrix producing large growth for any form of pivoting also generates large growth under reasonable assumptions. Finally, we demonstrate that GMRES-based iterative refinement can provide stable solutions to $Ax = b$ when large growth occurs in low precision LU factors, even when standard iterative refinement cannot.

A note on the universality of ESDs of inhomogeneous random matrices

In this short note, we extend the celebrated results of Tao and Vu, and Krishnapur on the universality of empirical spectral distributions to a wide class of inhomogeneous complex random matrices, by showing that a technical and hard-to-verify Fourier domination assumption may be replaced simply by a natural uniform anti-concentration assumption. Along the way, we show that inhomogeneous complex random matrices, whose expected squared Hilbert-Schmidt norm is quadratic in the dimension, and whose entries (after symmetrization) are uniformly anti-concentrated at $0$ and infinity, typically have smallest singular value $\Omega(n^{-1/2})$. The rate $n^{-1/2}$ is sharp, and closes a gap in the literature. Our proofs closely follow recent works of Livshyts, and Livshyts, Tikhomirov, and Vershynin on inhomogeneous real random matrices. The new ingredient is a couple of anti-concentration inequalities for sums of independent, but not necessarily identically distributed, complex random variables, which may also be useful in other contexts.

A Robust Wavenumber-Domain Superdirective Beamforming for Endfire Arrays

Among various array configurations, superdirective beamforming based on linear differential arrays has attracted much attention. It is well known that its directivity performance degrades at higher frequencies due to the limitation of finite difference approximation, and its robustness deteriorates due to the mismatch of practical systems. In this paper, we develop a wavenumber-domain superdirective beamforming for endfire arrays, an alternative way of phase mode beamforming that cannot be applied directly to linear arrays. Here, “wavenumber-domain” means the spatial derivatives are calculated in the wavenumber domain through multiplication operations. An important feature of proposed method is that it provides a novel way to obtain the higher order derivatives from the reconstructed sound field utilizing all sensors of array, which is valid when the Nyquist criterion is satisfied, resulting in a broadband frequency-independent beamformer. On the other hand, since the proposed $N\rm {th}$ -order superdirective beamformer has no upper limit of number of sensors, its robustness at lower frequencies can be enhanced through increasing the number of sensors and using an optimum truncation parameter which plays a role to remove small singular values sensitive to the mismatch of array. Although it is demonstrated that the proposed method and the weighted least-squares optimization method lead to almost identical performances, the main advantage of the proposed method is that an optimum truncation parameter determined by the experiment is adopted to increase the robustness of beamformers, where the high sensitivity of the higher eigenbeams to random errors is exploited.

Time Integration of Tree Tensor Networks

Dynamical low-rank approximation by tree tensor networks is studied for the data-sparse approximation of large time-dependent data tensors and unknown solutions to tensor differential equations. A time integration method for tree tensor networks of prescribed tree rank is presented and analyzed. It extends the known projector-splitting integrators for dynamical low-rank approximation by matrices and rank-constrained Tucker tensors and is shown to inherit their favorable properties. The integrator is based on recursively applying the low-rank Tucker tensor integrator. In every time step, the integrator climbs up and down the tree: it uses a recursion that passes from the root to the leaves of the tree for the construction of initial value problems on subtree tensor networks using appropriate restrictions and prolongations, and another recursion that passes from the leaves to the root for the update of the factors in the tree tensor network. The integrator reproduces given time-dependent tree tensor networks of the specified tree rank exactly and is robust to the typical presence of small singular values in matricizations of the connection tensors, in contrast to standard integrators applied to the differential equations for the factors in the dynamical low-rank approximation by tree tensor networks.

Efficient Pixel-Wise SVD Required for Image Processing Using the Color Line Feature

In color image processing based on mathematical optimization, a color-line image feature has been considered and many methods that give good results have been proposed. A color-line is a linear color distribution (correlation line) observed in a local image region, and is numerically represented by the sparsity of a data matrix generated from the neighboring pixel values. However, the calculation requires a lot of processing time because each data matrix is processed by inverse calculation or singular value decomposition (SVD) with some operations on the decomposed singular values. In this paper, in order to address this problem, we propose a method that can effectively compute SVD for each data matrix. Using the experimental knowledge that matrices obtained from neighboring regions (each centered at an adjacent pixel) are similar to each other, we intentionally design the SVD by using an iterative method (Arnoldi iteration), and propagate the converged singular vectors at a pixel to the next pixel as the initial vectors of the iteration. This propagation can drastically reduce the number of iterations required for convergence. Additionally, the singular values and vectors are obtained in descending order, which is advantageous when a matrix is reconstructed after reducing small singular values, so we can truncate the calculation when a singular value becomes lower than a threshold value. To show the effectiveness, we apply the proposed method to a denoising method, arranged structure-tensor total variation (ASTV), and show that the processing time is shortened by 95% compared to the naive method without losing numerical accuracy.

On the regularization effect of stochastic gradient descent applied to least-squares

We study the behavior of stochastic gradient descent applied to $\|Ax -b \|_2^2 \rightarrow \min$ for invertible $A \in \mathbb{R}^{n \times n}$. We show that there is an explicit constant $c_{A}$ depending (mildly) on $A$ such that $$ \mathbb{E} ~\left\| Ax_{k+1}-b\right\|^2_{2} \leq \left(1 + \frac{c_{A}}{\|A\|_F^2}\right) \left\|A x_k -b \right\|^2_{2} - \frac{2}{\|A\|_F^2} \left\|A^T A (x_k - x)\right\|^2_{2}.$$ This is a curious inequality: the last term has one more matrix applied to the residual $u_k - u$ than the remaining terms: if $x_k - x$ is mainly comprised of large singular vectors, stochastic gradient descent leads to a quick regularization. For symmetric matrices, this inequality has an extension to higher-order Sobolev spaces. This explains a (known) regularization phenomenon: an energy cascade from large singular values to small singular values smoothes.

Accelerating patch-based low-rank image restoration using kd-forest and Lanczos approximation

Patch-based low-rank approximation (PLRA) via truncated singular value decomposition is a powerful and effective tool for recovering the underlying low-rank structure in images. Generally, it first performs an approximate nearest neighbors (ANN) search algorithm to group similar patches into a collection of matrices with reshaping them as vectors. The inherent correlation among similar patches makes these matrices have a low-rank structure. Then the singular value decomposition (SVD) is used to derive a low-rank approximation of each matrix by truncating small singular values. However, the conventional implementation of patch-based low-rank image restoration suffers from high computational cost of the ANN search and full SVD. To address this limitation, we propose a fast approximation method that accelerates the computation of PLRA using multiple kd-trees and Lanczos approximation. The basic idea of this method is to exploit an index kd-tree built from patch samples of the observed image and several small kd-trees built from overlapping regions of the image to accelerate the search for similar patches, and apply the Lanczos bidiagonalization procedure to obtain a fast low-rank approximation of patch matrix without computing the full SVD. Experimental results on image denoising and inpainting tasks demonstrate the efficiency and accuracy of our method.

Determination of singular value truncation threshold for regularization in ill-posed problems

Appropriate regularization parameter specification is the linchpin for solving ill-posed inverse problems when regularization method is applied. This paper presents a novel technique to determine cut off singular values in the truncated singular value decomposition (TSVD) methods. Simple formulae are presented to calculate the index number of the singular value, beyond which all the smaller singular values and the corresponding vectors are truncated. The determination method of optimal truncation threshold is firstly theoretically inferred. Two-dimensional inverse problems processing Radon transform are then exemplified. Formulae to solve the problem with insufficient image resolution and projection angle number are derived by the currently proposed method. The results show that accuracy of the current method is similar to that of TSVD but with much superior efficiency. On the other hand, insufficiency in input data affects the output accuracy of the inverse solution, a least square method can be engaged to establish formulae calculating the truncation threshold. For an insufficient set of input data, the percentage difference between inversely reconstructed signal and TSVD reconstructed signal is about 3%. The current formulae offer reliable and more efficient approach to calculate the truncation threshold when TSVD is applied to solve inverse problems with known system characteristics.

On the smallest singular value in the class of invertible lower triangular (0,1) matrices

Given an n×n real symmetric matrix A, let λn(A) denote its smallest eigenvalue. Let Kn denote the set of all n×n invertible, lower triangular (0,1) matrices, and cn:=min{λn(YYt):Y∈Kn}. Then, cn is the smallest singular value in Kn. Hong and Loewy introduced cn as a mean to obtain inequalities involving eigenvalues of certain GCD (greatest common divisor) and LCM (least common multiple) matrices. Since then, cn has been used in many papers to obtain additional spectral inequalities for GCD and LCM matrices, and their generalizations. Due to its wide spread, it became important to obtain good bounds for cn. In this paper we obtain such bounds, and consequently determine the asymptotic behavior of cn, proving a conjecture of Kaarnioja. Moreover, we prove the uniqueness of the matrix Y∈Kn for which cn is attained, proving a conjecture of Altinisik, Keskin, Yildiz and Demirbüken.

An improved ridge estimation (IRE) method for troposphere water vapor tomography

Water vapor tomography technique is a research point in global navigation satellite system (GNSS) meteorology. However, GNSS tomography technique has an inherent issue of design matrix sparsity, which leads to the ill-posed problem in the inversion of normal equation. Singular value decomposition (SVD) method is commonly used to resolve this problem but performs poorly due to the large fluctuation in tomographic results caused by the small singular values. To overcome the above issue, this study proposes an improved ridge estimation (IRE) method, which only revises the relatively small singular values and retains the large singular values unchanged on the basis of the minimum deviation principle in the processing of regularization matrix. An adjustment coefficient is introduced to determine the node of small singular value with large influence on the tomographic result. Twelve GNSS stations from Hong Kong Satellite Positioning Reference Station Network (SatRef) were used to validate the performance of the proposed method over the period of day of year (Doy) 84–101, 2014. The established tomography model in SatRef was resolved using SVD and IRE methods, respectively. Statistical results show that the accuracy of average root mean square error (RMSE) of IRE method is improved by 16.7% compared with that of SVD method when the corresponding radiosonde is considered as the reference. Additionally, the comparisons of water vapor profile, relative error (RE) and the reconstructed SWV derived from IRE and SVD methods were also carried out and the numerical result indicates a good performance of the proposed IRE method. Such results verify that the applicability of proposed IRE method in this paper for GNSS water vapor tomography.

Smallest singular value and limit eigenvalue distribution of a class of non-Hermitian random matrices with statistical application

Suppose X is an N×n complex matrix whose entries are centered, independent, and identically distributed random variables with variance 1∕n and whose fourth moment is of order O(n−2). Suppose A is a deterministic matrix whose smallest and largest singular values are bounded below and above respectively, and z≠0 is a complex number. First we consider the matrix XAX∗−z, and obtain asymptotic probability bounds for its smallest singular value when N and n diverge to infinity and N∕n→γ,0<γ<∞. Then we consider the special case where A=J=[1i−j=1modn] is a circulant matrix. Using the above result, we show that the limit spectral distribution of XJX∗ exists when N∕n→γ,0<γ<∞ and describe the limit explicitly. Assuming that X represents a ℂN-valued time series which is observed over a time window of length n, the matrix XJX∗ represents the one-step sample autocovariance matrix of this time series. A whiteness test against an MA correlation model for this time series is introduced based on the above limit result. Numerical simulations show the excellent performance of this test.

A fault information-oriented weighted nuclear norm minimization method and its application to fault feature extraction in a rolling bearing

How to effectively extract fault features from noisy vibration signals is a key problem to be solved in bearing fault diagnosis. The difficulty here mainly lies in the fact that the components of interest are almost submerged by background noise and external interference. To accurately implement extraction of fault features, nuclear norm minimization (NNM) has been introduced to fault feature extraction due to its powerful denoising capability. However, NNM treats all singular values equally, which may result in components of interest with small singular values being abandoned, especially when a bearing defect is at its incipient stage. Therefore, to enhance the flexibility and effectiveness of NNM in handling practical problems, fault information-oriented weighted nuclear norm minimization (FIWNNM) is proposed for extracting bearing fault features, in which singular values are treated differently by assigning appropriate weights according to the physical meanings. Through the proposed FIWNNM, fault features can be drawn from a noisy signal, even when contaminated by strong noise and external interferences. The analysis results from both simulated and real measured signals are used as the verification of the proposed FIWNNM.

Comparing generic and vectorial nonlinear manoeuvring models and parameter estimation using optimal truncated least square support vector machine

An optimal truncated least square support vector machine (LS-SVM) is proposed for the parameter estimation of nonlinear manoeuvring models based on captive manoeuvring tests. Two classical nonlinear manoeuvring models, generic and vectorial models, are briefly introduced, and the prime system of SNAME is chosen as the normalization forms for the hydrodynamic coefficients. The optimal truncated LS-SVM is introduced. It is a robust method for parameter estimation by neglecting the small singular values, which contribute negligibly to the solutions and increase the parameter uncertainty. The parameter with a large uncertainty is sensitive to the noise in the data and have a poor generalization performance. The classical LS-SVM and optimal truncated LS-SVM are used to estimate the parameters, and the effectiveness of optimal truncated LS-SVM is validated. The parameter uncertainty for both nonlinear manoeuvring models is discussed. The generalization performance of the obtained numerical models is further tested against the validation set, which is completely left untouched in the training. The R2 goodness-of-fit criterion is used to demonstrate the accuracy of the obtained models.

Truncated least square support vector machine for parameter estimation of a nonlinear manoeuvring model based on PMM tests

A new version of least square support vector machine (LS-SVM), the truncated LS-SVM, is proposed to estimate the nondimensionalized hydrodynamic coefficients. Truncated LS-SVM is shown to be an efficient and robust method that avoids the costly matrix inversion operation in classical LS-SVM using the singular values decomposition. Meanwhile, the smaller singular values are neglected considering their negligible contribution to the solutions. In order to get a robust parameter estimation, the model simplification of the nonlinear manoeuvring model is carried out using a leave-one-out method, considering the trade-off between the parameter uncertainty and accuracy of the numerical model. The simplified manoeuvring model and the values of nondimensionalized hydrodynamic coefficients are presented. The coefficients are estimated using Planar motion mechanism (PMM) tests, which were carried out in a towing tank. The validation process is carried to validate the generalization performance of the obtained numerical model using the PMM test data.

A TV Forward-Looking Super-Resolution Imaging Method Based on TSVD Strategy for Scanning Radar

Because of the poor performance of the conventional total variation (TV) super-resolution imaging method in low signal-to-noise ratio (SNR) condition, a TV super-resolution imaging method based on the truncated singular value decomposition (TSVD) strategy is proposed. First, based on the regularization theory, the TV function is selected as the constraint term to construct objective function. Second, to solve the problem of noise amplification faced by the conventional TV method, this article reconstructs the objective function based on the TSVD strategy, which improves the antinoise performance by discarding small singular values of antenna convolution matrix. Finally, due to the nondifferentiable property of reconstructed objective function, this article utilizes the iterative reweighted norm (IRN) method. Since the influence of the noise is weakened by the TSVD strategy, the proposed method can achieve super-resolution imaging and contour preservation in low SNR condition. The simulation and experimental results demonstrate the effectiveness of the proposed method.

Nonlinear system modeling using self-organizing fuzzy neural networks for industrial applications

In this paper, a novel self-organizing fuzzy neural network with an adaptive learning algorithm (SOFNN-ALA) for nonlinear system modeling and identification in industrial processes is proposed. To efficiently enhance the generalization capability, the proposed SOFNN-ALA is designed by using both structure identification and parameter estimation simultaneously in the learning process. In the structure identification phase, the rule neuron with the highest neuronal activity will be split into two new rule neurons. Meanwhile, the redundant rule neurons with small singular values will be removed to simplify the network structure. In the parameter estimation phase, an adaptive learning algorithm (ALA), which is designed based on the widely used Levenberg-Marquardt (LM) optimization algorithm, is adopted to optimize the network parameters. The ALA-based learning algorithm can not only speed up the convergence speed but also enhance the modeling performance. Moreover, we carefully analyze the convergence of the proposed SOFNN-ALA to guarantee its successful practical application. Finally, the effectiveness and efficiency of the proposed SOFNN-ALA is validated by several examples. The experimental results demonstrate that the proposed SOFNN-ALA exhibits a better comprehensive performance than some other state-of-the-art SOFNNs for nonlinear system modeling in industrial applications. The source code can be downloaded from https://github.com/hyitzhb/SOFNN-ALA.git.