Smallest Singular Value Research Articles

Small Singular Values Can Increase in Lower Precision

Small Singular Values Can Increase in Lower Precision

A robust second-order low-rank BUG integrator based on the midpoint rule

Dynamical low-rank approximation has become a valuable tool to perform an on-the-fly model order reduction for prohibitively large matrix differential equations. A core ingredient is the construction of integrators that are robust to the presence of small singular values and the resulting large time derivatives of the orthogonal factors in the low-rank matrix representation. Recently, the robust basis-update & Galerkin (BUG) class of integrators has been introduced. These methods require no steps that evolve the solution backward in time, often have favourable structure-preserving properties, and allow for parallel time-updates of the low-rank factors. The BUG framework is flexible enough to allow for adaptations to these and further requirements. However, the BUG methods presented so far have only first-order robust error bounds. This work proposes a second-order BUG integrator for dynamical low-rank approximation based on the midpoint quadrature rule. The integrator first performs a half-step with a first-order BUG integrator, followed by a Galerkin update with a suitably augmented basis. We prove a robust second-order error bound which in addition shows an improved dependence on the normal component of the vector field. These rigorous results are illustrated and complemented by a number of numerical experiments.

Wegner estimate and upper bound on the eigenvalue condition number of non‐Hermitian random matrices

AbstractWe consider non‐Hermitian random matrices of the form , where is a general deterministic matrix and consists of independent entries with zero mean, unit variance, and bounded densities. For this ensemble, we prove (i) a Wegner estimate, that is, that the local density of eigenvalues is bounded by and (ii) that the expected condition number of any bulk eigenvalue is bounded by ; both results are optimal up to the factor . The latter result complements the very recent matching lower bound obtained by Cipolloni et al. and improves the ‐dependence of the upper bounds by Banks et al. and Jain et al. Our main ingredient, a near‐optimal lower tail estimate for the small singular values of , is of independent interest.

A dynamic light scattering inversion method based on regularization matrix reconstruction for flowing aerosol measurement.

Tikhonov regularization, or truncated singular value decomposition (TSVD), is usually used for dynamic light scattering (DLS) inversion of particles in suspension. The Tikhonov regularization method uses a regularization matrix to modify all singular values in the kernel matrix. The modification of large singular values cannot effectively reduce the variance of the estimated values but may introduce bias in the solution, resulting in poor disturbance resistance in the inversion results. The TSVD method, on the other hand, truncates all small singular values, which leads to the loss of particle size information during the inversion process. The shortcomings of the two methods mentioned above do not have a significant impact on the inversion of high signal-to-noise ratio data. However, compared to the classical DLS particle size inversion for non-flowing suspended particles, the DLS inversion of flowing aerosols is more significantly affected by noise, and the extraction of particle size information is more difficult due to the effect of flow velocity, resulting in worse inversion results with increasing aerosol flow velocity for both methods. To improve the accuracy of the particle size distribution (PSD) of flowing aerosols, we introduced a kernel matrix into the regularization matrix, and based on the principles of the two methods, the spectral information of the kernel matrix was utilized to make the modification of small singular values by the regularization matrix. Correspondingly, weak or no modification was made according to the values of large singular values to reduce the introduction of bias. The inversion results of simulated and measured data indicate that the reconstruction of the regularization matrix improves the anti-disturbance performance and avoids the loss of particle size information during the regularization inversion process, thereby significantly improving the PSD accuracy, which is affected by the dual effects of flow velocity and noise in the DLS measurement of flowing particles. The peak error and distribution error of the inversion results by reconstructing the regularization matrix are lower than those of Tikhonov regularization.

Tensor robust PCA with nonconvex and nonlocal regularization

Tensor robust principal component analysis (TRPCA) is a classical way for low-rank tensor recovery, which minimizes the convex surrogate of tensor rank by shrinking each tensor singular value equally. However, for real-world visual data, large singular values represent more significant information than small singular values. In this paper, we propose a nonconvex TRPCA (N-TRPCA) model based on the tensor adjustable logarithmic norm. Unlike TRPCA, our N-TRPCA can adaptively shrink small singular values more and shrink large singular values less. In addition, TRPCA assumes that the whole data tensor is of low rank. This assumption is hardly satisfied in practice for natural visual data, restricting the capability of TRPCA to recover the edges and texture details from noisy images and videos. To this end, we integrate nonlocal self-similarity into N-TRPCA, and further develop a nonconvex and nonlocal TRPCA (NN-TRPCA) model. Specifically, similar nonlocal patches are grouped as a tensor and then each group tensor is recovered by our N-TRPCA. Since the patches in one group are highly correlated, all group tensors have strong low-rank property, leading to an improvement of recovery performance. Experimental results demonstrate that the proposed NN-TRPCA outperforms existing TRPCA methods in visual data recovery. The demo code is available at https://github.com/qguo2010/NN-TRPCA.

Modal analysis of earthquake records for dams using stochastic subspace based on error analysis

AbstractSystem identification based on seismic data is crucial for updating numerical analyses and implementing health monitoring strategies for critical structures such as dams. However, most system identification methods have been developed with the assumption of white noise and stationarity of excitations, which is inconsistent with the nature of seismic data. As a result, the systems estimated from such methods are often associated with high uncertainties, making obtaining accurate and reliable results challenging. In this study, a new figure of merit (FOM) is proposed for tracking the ill‐conditioning of systems by analyzing the errors in stochastic subspace identification (SSI) through the inversion process. Unlike the condition number, which measures the ratio between the largest and smallest singular values, this FOM uses all singular values to assess the system's ill‐conditioning comprehensively. A semi‐automatic identification method based on SSI has been developed using the proposed FOM. The optimal dimensions of the Hankel matrix are obtained so that none of the system error components dominate. Frequency/damping clustering is used to overcome the identified system's over‐determination and remove outliers. Finally, the complexity of the modal shapes resulting from clustering is examined to validate the structural modal characteristics. The procedure was validated using seismic data from the Finite Element Model of Koyna Dam. It identified the first four modal frequencies of the structure with less than 2% error rate. The average error in estimating damping ratios was below 20%. The algorithm was then used to analyze the seismic monitoring data of Pacoima Dam for the Chino Hill 2008 and Granada Hills 2020 earthquakes. Despite the closely spaced Pacoima Dam modes, the study demonstrated excellent accuracy and robust methodology performance. By distinguishing the structural modes related to the dam body from other modes of the “dam‐foundation‐reservoir” system, the proposed method helps to determine the existing model's nonlinearity.

Understanding How Orthogonality of Parameters Improves Quantization of Neural Networks.

We analyze why the orthogonality penalty improves quantization in deep neural networks. Using results from perturbation theory as well as through extensive experiments with Resnet50, Resnet101, and VGG19 models, we mathematically and experimentally show that improved quantization accuracy resulting from orthogonality constraint stems primarily from reduced condition numbers, which is the ratio of largest to smallest singular values of weight matrices, more so than reduced spectral norms, in contrast to the explanations in previous literature. We also show that the orthogonality penalty improves quantization even in the presence of a state-of-the-art quantized retraining method. Our results show that, when the orthogonality penalty is used with quantized retraining, ImageNet Top5 accuracy loss from 4- to 8-bit quantization is reduced by up to 7% for Resnet50, and up to 10% for Resnet101, compared to quantized retraining with no orthogonality penalty.

Oblique projection for scalable rank-adaptive reduced-order modelling of nonlinear stochastic partial differential equations with time-dependent bases

Time-dependent basis reduced-order models (TDB ROMs) have successfully been used for approximating the solution to nonlinear stochastic partial differential equations (PDEs). For many practical problems of interest, discretizing these PDEs results in massive matrix differential equations (MDEs) that are too expensive to solve using conventional methods. While TDB ROMs have the potential to significantly reduce this computational burden, they still suffer from the following challenges: (i) inefficient for general nonlinearities, (ii) intrusive implementation, (iii) ill-conditioned in the presence of small singular values and (iv) error accumulation due to fixed rank. To this end, we present a scalable method for solving TDB ROMs that is computationally efficient, minimally intrusive, robust in the presence of small singular values, rank-adaptive and highly parallelizable. These favourable properties are achieved via oblique projections that require evaluating the MDE at a small number of rows and columns. The columns and rows are selected using the discrete empirical interpolation method (DEIM), which yields near-optimal matrix low-rank approximations. We show that the proposed algorithm is equivalent to a CUR matrix decomposition. Numerical results demonstrate the accuracy, efficiency and robustness of the new method for a diverse set of problems.

Boundary between noise and information applied to filtering neural network weight matrices.

Deep neural networks have been successfully applied to a broad range of problems where overparametrization yields weight matrices which are partially random. A comparison of weight matrix singular vectors to the Porter-Thomas distribution suggests that there is a boundary between randomness and learned information in the singular value spectrum. Inspired by this finding, we introduce an algorithm for noise filtering, which both removes small singular values and reduces the magnitude of large singular values to counteract the effect of level repulsion between the noise and the information part of the spectrum. For networks trained in the presence of label noise, we indeed find that the generalization performance improves significantly due to noise filtering.

An interval perturbation method for singular value decomposition (SVD) with unknown-but-bounded (UBB) parameters

The singular value decomposition (SVD) is a crucial method with successful practical applications. However, the quantification of uncertainties in SVD perturbation is challenging. To address this issue, this study introduces an interval perturbation method for SVD with unknown-but-bounded (UBB) parameters. Unlike probabilistic approaches that require statistical data on uncertain parameters, this method only necessitates the bounds of uncertain parameters. By using non-probabilistic theory, the proposed method can provide accurate and fast estimation of singular values and vectors. The paper provides a detailed derivation process for obtaining the interval bounds of singular values and vectors using the interval perturbation method. The subinterval method is applied to improve the estimation precision. The effectiveness of the proposed method is demonstrated through four numerical examples and an application example. The robustness of the proposed method is verified by various levels of uncertainties, different cases of subintervals, different matrix scales, close singular values, rectangular matrices, and small but nonzero singular values. The proposed method can be used in engineering numerical calculation inverse problem fields that require SVD with small uncertainties, including dynamic identification, image processing, and signal processing. Therefore, the superiority of the proposed method in accurately and quickly estimating singular values and vectors with uncertainties can be applied in the aforementioned research fields.

Model order reduction based on low-rank decomposition of the cross Gramian

This paper considers model order reduction based on low-rank decomposition of the cross Gramian for linear systems. The proposed approach uses the low-rank factors of the cross Gramian to generate approximate balanced model for the large-scale system. Then, the reduced-order model is obtained by truncating the states corresponding to the small approximate Hankel singular values. The low-rank factors are directly constructed from the expansion coefficients of impulse responses in the space spanned by Legendre polynomials by solving block tridiagonal linear systems. In contrast to balanced truncation related approaches which require to solve Sylvester equation to compute the full cross Gramian, the proposed method needs to solve sparse linear equations, and only one singular value decomposition is applied to a low-dimensional matrix. The stability preservation of the reduced model is briefly discussed. And in combination with the dominant subspace projection method, the reduction procedure is modified to alleviate the shortcoming, which may unexpectedly lead to unstable systems even though the original one is stable. Finally, numerical experiments are given to demonstrate the effectiveness of the proposed methods.

Tikhonov regularization for the deconvolution of capacitance from the voltage–charge response of electrochemical capacitors

The capacitance of capacitive energy storage devices cannot be directly measured, but can be estimated from the applied input and measured output signals expressed in the time or frequency domains. Here the time-domain voltage–charge relationship of non-ideal electrochemical capacitors is treated as an ill-conditioned convolution integral equation where the unknown capacitance kernel function is to be found. This comes from assuming a priori that in the frequency domain the charge is equal to the product of capacitance by voltage, which is in line with the definition of electrical impedance. The computation of a stable solution to this problem particularly when dealing with experimental data is highly sensitive to noise as it may lead to an oscillating result even in the presence of small errors in the measurements. In this work, the problem is treated using Tikhonov’s regularization method, where a degree of damping is added to each singular value decomposition (SVD) component of the solution, thus effectively filtering out the components corresponding to the small singular values. The regularized time-domain capacitance of a commercial electric double-layer capacitor is found to be in good agreement with the frequency-to-time transformed capacitance obtained from impedance spectroscopy.

Three dimensional seismic data reconstruction based on truncated nuclear norm

The rank reduction method is widely used to reconstruct three dimensional (3D) seismic data. The traditional multichannel singular spectrum analysis (MSSA) utilizes truncated singular value decomposition (TSVD) to approximately solve the rank function of the block Hankel structure matrix constructed by frequency slices of seismic data. However, the TSVD algorithm discards all nonzero singular values except a few largest singular values and ignores the useful seismic information in small nonzero singular values. To further utilize the seismic data information contained in these small singular values, this paper proposes a method to recover 3D seismic data with truncated nuclear norm (TNN). The proposed method imposes different constraints on large singular values and small singular values. Essentially, the TNN is closer to the rank function than the TSVD. Finally, the proposed model is addressed by the alternating direction method of multipliers, and closed-form solutions are used to update the optimization variables. The experimental results demonstrate that the proposed method achieves superior reconstruction results than the traditional MSSA method.

Differential privacy protection algorithm for network sensitive information based on singular value decomposition

In order to reduce the risk of data privacy disclosure and improve the effect of information privacy protection, a differential privacy protection algorithm for network sensitive information based on singular value decomposition is proposed. TF-IDF method is used to extract network sensitive information text. By comparing the word frequency of network sensitive information, high word frequency word elements in network information content are collected to obtain the mining results of network sensitive information text. According to the decision tree theory, the equal difference privacy budget allocation mechanism is improved to achieve equal difference privacy budget allocation. By discarding some small singular values and corresponding spectral vectors, the data can be disturbed, and the availability of the original data can be retained, so that it can truly represent the original data set structure. According to the results of equal difference privacy budget allocation and singular value decomposition disturbance, the data of high-dimensional network graph is reduced by random projection, singular value decomposition is performed on the reduced data, and Gaussian noise is added to the singular value. Finally, the matrix to be published is generated through the inverse operation of singular value decomposition to achieve differential privacy protection of network sensitive information. The experimental results show that the privacy protection quality of this algorithm is high and the data availability is effectively improved.

Multi-focus image fusion based on gradient tensor HOSVD

Multi-focus image fusion is the process of fusing images of the same scene with different focus ranges. However, pixel misclassification and artifacts caused by noise have been unavoidable problems in fusion methods. To overcome this problem, we propose gradient tensor high-order singular value decomposition (HOSVD) to fuse multi-focus images. The eight-direction Sobel gradient operator is used to obtain second-order gradients in different directions. Tensor-based information processing can effectively represent high-dimensional information and extract structural information from multi-focus images. Therefore, second-order gradient matrices of different images are formed into tensors. The gradient tensor of the images is transformed by HOSVD to obtain the core tensor and factor matrixes. The images are better represented by multiplying the core tensor with the factor matrix corresponding to the direction of the image. In the core tensor, the larger singular values represent basic information and the small singular values high-frequency information. The noise generally belongs to high-frequency information, so we select the larger singular values in the core tensor as activity level variables of the image to reduce noise. Experimental results demonstrate that the proposed fusion method can effectively reduce noise effects and achieve state-of-the-art fusion performance in both objective metrics and visual quality.

Structure-preserving model order reduction for K-power bilinear systems via Laguerre functions

ABSTRACT This paper presents a series of structure-preserving model order reduction algorithms for K-power bilinear systems via Laguerre functions. The method first aims to rewritten the K-power bilinear system as a general bilinear system and calculate the approximate low-rank factors of the cross Gramian of the bilinear system by combining the idea of Laguerre functions expansion of the matrix exponential function. After that, the approximate balanced system of the K-power bilinear system is constructed by the corresponding projection transformation of each subsystem. In order to achieve the purpose of model order reduction, the states with smaller singular values are then truncated, so as to further obtain the reduced order model. For this approach, there is a disadvantage that unstable systems may be generated although the original one is stable. To alleviate the inadequacies of this approach, we have improved the model reduction procedure, which is based upon the dominant subspace projection method. In addition, we also carried out a correlation analysis on the stability of improved algorithms. Finally, numerical experiments are employed to substantiate the effectiveness of the presented algorithms.

An Optimization Strategy to Position CHIEF Points in Boundary-Element Acoustic Problems

The use of boundary elements in the analysis of exterior acoustic problems poses challenges at specific frequencies, since fictitious eigenfrequencies may arise at the internal resonances of cavities, leading to inaccurate results or even unstable behavior. To filter out these fictitious eigenfrequencies, a scheme based on the combined Helmholtz integral equation formulation (CHIEF) can be used to prevent the so-called non-uniqueness problem, although it requires additional equations and points. The BEM formulation final accuracy will, however, depend on the correct choice of these points. Here, a strategy to help in defining good approximations for the position and number of such points is proposed, based on an optimization process which maximizes the system matrix’s smallest singular value. The accuracy of the method for exterior radiation problems is investigated using different examples. With low computational cost and simple implementation, the two proposed algorithms automatically circumvent the non-uniqueness problem, aiding the implementation of more stable BEM codes.

Rank-Adaptive Time Integration of Tree Tensor Networks

A rank-adaptive integrator for the approximate solution of high-order tensor differential equations by tree tensor networks is proposed and analyzed. In a recursion from the leaves to the root, the integrator updates bases and then evolves connection tensors by a Galerkin method in the augmented subspace spanned by the new and old bases. This is followed by rank truncation within a specified error tolerance. The memory requirements are linear in the order of the tensor and linear in the maximal mode dimension. The integrator is robust to small singular values of matricizations of the connection tensors. Up to the rank truncation error, which is controlled by the given error tolerance, the integrator preserves norm and energy for Schrödinger equations, and it dissipates the energy in gradient systems. Numerical experiments with a basic quantum spin system illustrate the behavior of the proposed algorithm.

Combined empirical mode decomposition and phase space reconstruction based psychologically stressed and non-stressed state classification from cardiac sound signals

The psychological stress and associated mental health conditions causes high socioeconomic impacts on society and the onset of pandemic worsened the situation making it imperative to timely detect psychological stress. This paper presents a novel framework for using a combination of empirical mode decomposition (EMD) and phase space reconstruction (PSR) analysis to capture non-linear, non-stationary dynamics of cardiac sound signals acquired from fifty-four healthy male adults for detecting psychological stress. The time interval among successive S1 peaks of acquired cardiac sound signals is extracted to obtain Interbeat Interval signal used for decomposition to Intrinsic Mode Functions (IMFs) using EMD technique. Thereafter, the feature vectors namely- largest singular value, smallest singular value from two-dimensional PSRs of IMFs and the mean value of Euclidean distance, standard deviation of Euclidean distance from three-dimensional PSRs are extracted to detect psychologically stressed state. The non-parametric Kruskal-Wallis statistical test is applied to select statistically significant features that are fed to Decision Tree, Naïve Bayes and Support Vector Machine classifiers for classifying stressed and non-stressed state with fivefold cross-validation to make it a reliable system. The average accuracy, sensitivity and specificity achieved is 97.14%, 99.8% and 94% respectively using SVM and Radial Basis Function kernel function. The proposed framework performed better on the dataset in comparison to ratio of low-frequency to high-frequency (LF/HF) power parameter of Electrocardiography signal. The use of easy to acquire, cost-effective cardiac sound signals for detecting psychological stress makes proposed framework feasible for rural healthcare centres of developing economies, homecare and telemedicine.

Rank minimization via adaptive hybrid norm for image restoration

Rank minimization methods have achieved promising performance in various image processing tasks. However, there are still two challenging problems in the existing works. One is that most of the current methods only regularize singular values by using a single l1-norm, such as the well-known nuclear norm minimization (NNM) and the weighted nuclear norm minimization (WNNM). Consequently, many small singular values are shrunk to zero, which is unbeneficial for restoring image details. The other is that how to adaptively evaluate the importance of each singular value is still a suspending problem. In this paper, we propose a novel rank minimization method, namely adaptive hybrid norm minimization (AHNM) model, to solve the above problems. Specifically, for each singular value, we employ l2-norm to compensate for l1-norm, and introduce a significance factor to assess its importance adaptively. More importantly, we show that closed-form solutions for all subproblems can be derived simply by using alternating optimization. With the aid of the proposed AHNM model, we further develop a general yet effective image restoration algorithm based on the nonlocal self-similarity (NSS) of images. Numerous experimental results demonstrate that the proposed AHNM model consistently outperforms many state-of-the-art restoration methods, including model-based methods and deep learning-based methods.