High-dimensional Covariance Matrix Research Articles

Change point detection in high dimensional covariance matrix using Pillai’s statistics

AbstractThis research proposes a method to test and estimate change points in the covariance structure of high-dimensional multivariate series data. Our method uses the trace of the beta matrix, known as Pillai’s statistics, to test the change in covariance matrix at each time point. We study the asymptotic normality of Pillai’s statistics for testing the equality of two covariance matrices when both sample size and dimension increase at the same rate. We test the existence of a single change point in a given time period using Cauchy combination test, the test using an weighted sum of Cauchy transformed p-values, and estimate the change point as the point whose statistic is the greatest. To test and estimate multiple change points, we use the idea of the wild binary segmentation and repeatedly apply the procedure for a single change point to each segmented period until no significant change point exists. We numerically provide the size and power of our method. We finally apply our procedure to finding abnormal behavior in the investment of a private equity fund.

Efficient and Robust Adaptive Beamforming Based on Coprime Array Interpolation

Unlike uniform linear arrays (ULAs), coprime arrays require fewer physical sensors yet provide higher degrees of freedom (DOF) and larger array apertures. However, due to the existence of “holes” in the differential co-array, the target detection performance deteriorates, especially in adaptive beamforming. To address these challenges, this paper proposes an efficient and robust adaptive beamforming algorithm leveraging coprime array interpolation. The algorithm eliminates unwanted signals and uses the Gauss–Legendre quadrature method to reconstruct an Interference-plus-Noise Covariance Matrix (INCM), thereby obtaining the beamforming coefficients. Unlike previous techniques, we utilize a virtual interpolated ULA to expand the aperture, enabling the acquisition of a high-dimensional covariance matrix. Additionally, a projection matrix is constructed to eliminate unwanted signals from the received data, greatly enhancing the accuracy of INCM reconstruction. To address the high computational complexity of integral operations used in most INCM reconstruction algorithms, we propose an approximation based on the Gauss–Legendre quadrature, which reduces the computational load while maintaining accuracy. This algorithm avoids the array aperture loss caused by using only the ULA segment in the difference co-array and improves the accuracy of INCM reconstruction. Simulation and experimental results show that the performance of the proposed algorithm is superior to the compared beamformers and is closer to the optimal beamformer in various scenarios.

A double sampling ridge penalized likelihood ratio control charting method with variable sample size for Phase II monitoring of high-dimensional covariance matrix

Today, manufacturers face two challenges: the financial limitation of sampling and the large number of study quality characteristics. In such situations, the problem dimension is usually larger than the sample size, which is known in the literature of statistical quality control as a high dimensional process. The use of conventional multivariate charts to monitor the covariance matrix of such processes is not possible for two reasons: (1) the non-reversibility of the sample covariance matrix, (2) the possibility of the occurrence of sparse shifts in which only a few elements of the covariance matrix are affected. This paper focuses on Phase II monitoring of the covariance matrix of high-dimensional processes based on integrating the ridge penalized likelihood ratio (RPLR) statistic and the combined DS-VSS sampling method, in which DS and VSS features are used concurrently. The performance of the proposed DS-VSS RPLR control chart is compared with the RPLR one in terms of three metrics of the average run length, standard deviation of the run length, and expected value of sample size under three types of joint diagonal/off-diagonal, diagonal and off-diagonal shift patterns. The results show that the DS-VSS scheme is more sensitive than the RPLR in detecting both sparse and non- sparse changes.

Energy and reserve procurement in integrated electricity and heating system: A high-dimensional covariance matrix approach based on stochastic differential equations

The integration of stochastic renewable energy sources into energy systems presents challenges due to flexibility insufficiency and uncertainty modeling inaccuracy. To address these challenges, this paper explores a high-dimensional covariance matrix approach based on stochastic differential equations (SDEs) for energy and reserve co-dispatch in integrated electricity and heating systems. By capturing the long-term correlation of wind power forecast errors, the SDE model simultaneously models point forecasts and high-dimensional positive-definite covariance matrices, resulting in a compact uncertainty set. To ensure computational efficiency, we employ duality theory to transform the stochastic co-dispatch problem with the high-dimensional uncertainty set into deterministic convex programming. Additionally, a virtual heat storage model is introduced for the district heating network, leveraging the thermal inertia of the network to enhance reserve capacity. Simulations validate the calibration and sharpness of the proposed high-dimensional uncertainty set. The results demonstrate that co-dispatch using the SDE-based uncertainty set reduces operational costs by 4.9% compared to a set that does not consider the long-term correlation of wind power, under a 95% coverage rate. Moreover, the storage capabilities of district heating pipelines and smart buildings provide cost-free reserves, saving the operational costs of the two test systems by 2.8% and 1.3%, respectively.

ON BLOCKWISE AND REFERENCE PANEL-BASED ESTIMATORS FOR GENETIC DATA PREDICTION IN HIGH DIMENSIONS.

Genetic prediction holds immense promise for translating genetic discoveries into medical advances. As the high-dimensional covariance matrix (or the linkage disequilibrium (LD) pattern) of genetic variants often presents a block-diagonal structure, numerous methods account for the dependence among variants in predetermined local LD blocks. Moreover, due to privacy considerations and data protection concerns, genetic variant dependence in each LD block is typically estimated from external reference panels rather than the original training data set. This paper presents a unified analysis of blockwise and reference panel-based estimators in a high-dimensional prediction framework without sparsity restrictions. We find that, surprisingly, even when the covariance matrix has a block-diagonal structure with well-defined boundaries, blockwise estimation methods adjusting for local dependence can be substantially less accurate than methods controlling for the whole covariance matrix. Further, estimation methods built on the original training data set and external reference panels are likely to have varying performance in high dimensions, which may reflect the cost of having only access to summary level data from the training data set. This analysis is based on novel results in random matrix theory for block-diagonal covariance matrix. We numerically evaluate our results using extensive simulations and real data analysis in the UK Biobank.

A Bayesian Structural Modal Updating Method Based on Sparse Grid and Ensemble Kalman Filter

This study presents a sparse grid interpolation and ensemble Kalman filter (EnKF)-based Markov Chain Monte Carlo (MCMC) method (SG-EnMCMC). Initiating with the formulation of a recursive equation for the state space vector, derived from the structural dynamic equation, this study adopts a dimensionality reduction strategy. This approach involves a separation of physical parameters and the state space vector. The acquisition of physical parameters is accomplished through sampling, utilizing sample moments to substitute population moments, thereby mitigating the need for computationally high-dimensional covariance matrix calculations. To further streamline the recursive equation of the state space vector, a sparse grid method is employed for interpolation. This step simplifies the process while ensuring superior accuracy compared to the Extended Kalman Filter (EKF) and Unscented Kalman Filter (UKF). Subsequent to this, acceptance rates and the final parameter posterior distribution within the MCMC framework are derived. The efficiency of the proposed method is assessed through validation in two shaking table experiments.

Target selection in shrinkage estimation of covariance matrix: A structural similarity approach

The shrinkage estimator of a high-dimensional covariance matrix relies on a preassigned target matrix during data processing. This paper provides an adaptive approach for selecting the optimal Toeplitz target matrix. We discover a sufficient and necessary condition for characterizing the two kinds of target matrices with the Toeplitz structure, and we propose an adaptive selection algorithm by measuring the similarity between the data and the Toeplitz structure. Numerical simulations and an empirical study on monetary funds verify the effectiveness of the selection approach.

Adaptive Tests for Bandedness of High-dimensional Covariance Matrices

Estimation of the high-dimensional banded covariance matrix is widely used in multivariate statistical analysis. To ensure the validity of estimation, we aim to test the hypothesis that the covariance matrix is banded with a certain bandwidth under the high-dimensional framework. Though several testing methods have been proposed in the literature, the existing tests are only powerful for some alternatives with certain sparsity levels, whereas they may not be powerful for alternatives with other sparsity structures. The goal of this paper is to propose a new test for the bandedness of high-dimensional covariance matrix, which is powerful for alternatives with various sparsity levels. The proposed new test also be used for testing the banded structure of covariance matrices of error vectors in high-dimensional factor models. Based on these statistics, a consistent bandwidth estimator is also introduced for a banded high dimensional covariance matrix. Extensive simulation studies and an application to a prostate cancer dataset from protein mass spectroscopy are conducted for evaluating the effectiveness of the proposed adaptive tests blue and bandwidth estimator for the banded covariance matrix.

Sub-Gaussian High-Dimensional Covariance Matrix Estimation under Elliptical Factor Model with 2 εth Moment

Sub-Gaussian High-Dimensional Covariance Matrix Estimation under Elliptical Factor Model with 2 εth Moment

High-Dimensional Covariance Matrix Estimation: Shrinkage Toward a Diagonal Target

High-Dimensional Covariance Matrix Estimation: Shrinkage Toward a Diagonal Target

Online portfolio selection with predictive instantaneous risk assessment

Online portfolio selection (OPS) has received increasing attention from machine learning and quantitative finance communities. Despite their effectiveness, the pioneering OPS methods have several key limitations. First, price predictions are usually based on predetermined trends, which is inadequate for a fast-changing market patterns. Second, each asset is treated individually, ignoring the pervading relevance among the assets. Third, the risk terms are usually missing or inappropriate in optimizations. This paper proposes a novel OPS method, namely, the online low-dimension ensemble method, to overcome the limitations. Motivated by the stylized facts for the co-movements of assets, the financial market is regarded as a high-dimensional dynamical system (HDS), and a large number of low-dimensional subsystems (LDSs) are randomly generated from the HDS to extract the correlation information among the assets. The assets’ price predictions are first made using these LDSs and then aggregated to formulate the final prediction using ensemble learning techniques. Thanks to the particular merits brought by our predicting scheme, we also develop a novel high-dimensional covariance matrix estimation/prediction method for short-term data, efficiently assessing the instantaneous risk of the projected portfolios. Compared with state-of-the-art methods, our approach obtains more accurate predictions as the correlation information is fully exploited. With the predictive instantaneous risk assessment, a more appropriate optimization problem is proposed, substantially improving the OPS setting and leading to significantly better investment performance. Therefore, this study develops a flexible and promising approach to learning fast-changing market patterns and demonstrates that the high-dimensional feature of the market is a crucial information source for financial modeling with short-term data rather than a barrier in the conventional sense. Extensive experiments on real-world datasets are conducted to illustrate our method further.

USV Parameter Estimation: Adaptive Unscented Kalman Filter-Based Approach

In this article, a new kind of adaptive unscented Kalman filter (AUKF) is proposed to deal with the parameter estimation problem for a class of nonlinear unmanned surface vessel (USV) models with unknown statistical characteristics of process noises. Specifically, the considered parameter estimation problem is first transformed into the state estimation problem by extending eighteen parameters and three unknown inputs into augmented states for the USVs. With the help of such a transformation, the unknown inputs including disturbances and modeling errors are estimated effectively, and employed to construct the estimators such that the effect of these unknown inputs on parameter estimation can be significantly suppressed. Under the condition that the structure of the covariance matrix of the process noise is available, an adaptive law is further designed to estimate such a high-dimensional covariance matrix where the covariance estimation errors can be reduced. Finally, the proposed estimation approach is verified via performing the practical experiment as well as numerical simulations.

Hypothesis testing on compound symmetric structure of high-dimensional covariance matrix

Testing the population covariance matrix is an important topic in multivariate statistical analysis. Owing to the difficulty of establishing the central limit theorem for the test statistic based on sample covariance matrix, in most of the existing literature, it is assumed that the population covariance matrix is bounded in spectral norm as the dimension tends to infinity. Four test statistics are proposed for testing the compound symmetric structure of the population covariance matrix, which has an unbounded spectral norm as the dimension tends to infinity. Three of these tests maintain high power against different kinds of dense alternatives. The asymptotic properties of these statistics are constructed under the null hypothesis and a specific alternative hypothesis. Moreover, extensive simulation studies and an analysis of real data are conducted to evaluate the performance of our proposed tests. The simulation results show that the proposed tests outperform existing methods.

A Best Linear Empirical Bayes Method for High-Dimensional Covariance Matrix Estimation

Covariance matrix estimation plays a significant role in both in the theory and practice of portfolio analysis and risk management. This paper deals with the available data prior to developing a factor model to enhance covariance matrix estimation. Our work has two main outcomes. First, for a general linear model with unknown prior parameters, a class of best linear empirical Bayes estimators is established through two kinds of architectures to improve the estimation accuracy by utilizing additional data prior. The theoretical results indicate two key points: the proposed estimators are equivalent to the linear minimum mean-square error estimator when complete or sufficient partial data prior are provided; and the proposed estimators perform better than the optimal weighted least squares method, which ignores the data prior in each situation. Second, the proposed estimators are used for calculating a high-dimensional covariance matrix through factor models. The numerical example and the simulation results verify the effectiveness of our methods.

High-Dimensional Covariance Estimation via Constrained Lq-Type Regularization

High-dimensional covariance matrix estimation is one of the fundamental and important problems in multivariate analysis and has a wide range of applications in many fields. In practice, it is common that a covariance matrix is composed of a low-rank matrix and a sparse matrix. In this paper we estimate the covariance matrix by solving a constrained Lq-type regularized optimization problem. We establish the first-order optimality conditions for this problem by using proximal mapping and the subspace method. The proposed stationary point degenerates to the first-order stationary points of the unconstrained Lq regularized sparse or low-rank optimization problems. A smoothing alternating updating method is proposed to find an estimator for the covariance matrix. We establish the convergence of the proposed calculation method. The numerical simulation results show the effectiveness of the proposed approach for high-dimensional covariance estimation.

A generalized multiple dependent state sampling chart based on ridge penalized likelihood ratio for high-dimensional covariance matrix monitoring

Online monitoring of high-dimensional processes variability in which the number of variables is larger than the sample size is a challenging issue for quality practitioners because the sample covariance matrix is not invariable. To deal with this challenge, a generalized multiple dependent state sampling (GMDS) chart based on ridge penalized likelihood ratio (RPLR) statistic is developed for Phase II monitoring of multivariate process variability under high-dimensional setting. The developed control chart benefits from three advantages: (1) departing from the conventional covariance matrix charts, it can be efficiently employed for both spars and non-spars covariance matrices; (2) it is able to detect spars shift patterns in which only a few covariance matrix elements are deviated from their nominal values; and (3) it outperforms the detectability of the RPLR chart in terms of average run length (ARL) and standard deviation of run length (SDRL). The performance of RPLR, MDS-RPLR, and GMDS-RPLR charts are compared using extensive simulation studies by considering different diagonal and/or off-diagonal covariance matrix disturbance. Moreover, sensitivity analysis are provided to analyze how the number of process variables and GMDS parameter affect the run length properties of the developed chart.

Sparsistent filtering of comovement networks from high-dimensional data

Network filtering is a technique to isolate core subnetworks of large and complex interconnected systems, which has recently found many applications in financial, biological, physical and technological networks among others. We introduce a new technique to filter large dimensional networks arising out of dynamical behavior of the constituent nodes, exploiting their spectral properties. As opposed to the well known network filters that rely on preserving key topological properties of the realized network, our method treats the spectrum as the fundamental object and preserves spectral properties. Applying asymptotic theory of high-dimensional covariance matrix estimation, we show that the proposed filter can be tuned to interpolate between zero filtering to maximal filtering that induces sparsity via thresholding, while having the least spectral distance from a consistent (non-)linear shrinkage estimator. We demonstrate the application of our proposed filter by applying it to covariance networks constructed from financial data, to extract core subnetworks embedded in full networks.

Development Model of College English Education Service Talents Based on Nonlinear Random Matrix

The development of artificial intelligence makes people’s life and work easier and more effective, and computer-based online exams and marking not only improve students’ learning efficiency but also reduce the pressure of teachers’ marking work. For objective questions, marking has gone from manual marking to cursor reader marking to computerized character matching, and the correct rate of marking has soared to 100%; for subjective questions, foreign systems such as PEG and E-rater have been used, and domestic systems such as those using English large corpus similarity matching and those based on natural language understanding using intelligent algorithms have been used for marking. Most of these systems are based on some shallow linguistic features such as rules and LSAs for marking, and there is no deep perception of English language sense. Although the current intelligent marking systems have made a lot of achievements, they do not fundamentally solve the problem of the rationality of intelligent marking of subjective questions. In this article, we propose a regularized discriminant analysis algorithm with good estimation of the mean, and a dimensionality reduction algorithm for high-dimensional missing data by using the relevant research results of random matrix theory to address the problems of traditional machine learning methods in high-dimensional data analysis. Although the linear discriminant analysis algorithm performs well in solving many practical problems, it works poorly in dealing with high-dimensional data. The specific analysis is as follows: in terms of age characteristics, the mobile population under the age of 35 has a significant preference for urban consumer comfort, and it increases with age, peaking at the stage of 30–35 years old and then decreasing rapidly. For this reason, a regularized discriminant analysis algorithm based on random matrix theory is proposed. First, a good estimate of the high-dimensional covariance matrix is made by the nonlinear shrinkage method or the eigenvalue interception method, respectively; then, the estimated high-dimensional covariance matrix is used to calculate the discriminant function values and perform the classification. The classification experiments conducted on simulated and real datasets show that the proposed algorithm is not only more widely applicable but also has a high correct classification rate.

Empirical Analysis of the Impact of High and New Technology on the Export of Traditional Cultural Industry Based on Random Matrix Theory

Accurate estimation of covariance matrix is the basis of effective development of high-dimensional optimal portfolio. Random matrix theory provides an effective means to improve the estimation of high-dimensional covariance matrix. Based on the empirical research on the export excess return rate of traditional cultural industry, the result is a combination with higher precision and lower risk. Based on relevant data, this paper constructs a stochastic matrix theoretical model to analyze the impact of human capital, new fixed asset investment, independent innovation and technology purchase, financing sources, and other factors on high-tech industry export. Based on the results of empirical analysis, some policy suggestions are put forward, such as increasing r&d investment, improving enterprises’ innovation ability, and introducing core scientific and technological talents to promote the export scale growth of high-tech industries. Model comparative analysis is made on the influence of traditional industries’ concentrated export, foreign trade environment, and cost advantage on the export of high-tech products. Through comparative analysis of the influencing factors of the export of high-tech products in the two periods before and after the export crisis of traditional culture, the new changes of the influencing factors of the export of high-tech products are analyzed. In addition, through the change of the internal and external environment of the current high-tech product export, the reasons for the new change of the influencing factors are discussed.

High-dimensional correlation matrix estimation for general continuous data with Bagging technique

High-dimensional covariance matrix estimation plays a central role in multivariate statistical analysis. It is well-known that the sample covariance matrix is singular when the sample size is smaller than the dimension of the variable, but the covariance estimate must be positive-definite. This motivates some modifications of the sample covariance matrix to preserve its efficient estimation of pairwise covariance. In this paper, we modify the sample correlation matrix using the Bagging technique. The proposed Bagging estimator is flexible for general continuous data. Under some mild conditions, we show theoretically that the Bagging estimator can ensure positive-definiteness with probability one in finite samples. We also prove the consistency of the bootstrap estimator of Pearson correlation and the consistency of our Bagging estimator when the dimension p is fixed. Simulation results and a real application are provided to demonstrate that our method strikes a better balance between RMSE and likelihood, and is more robust, than other existing estimators.