Classical Statistical Theory Research Articles

Understanding Surprising Generalization Phenomena in Deep Learning

Deep learning has exhibited a number of surprising generalization phenomena that are not captured by classical statistical learning theory. This talk will survey some of my work on the theoretical characterizations of several such intriguing phenomena: (1) Implicit regularization: A major mystery in deep learning is that deep neural networks can often generalize well despite their excessive expressive capacity. Towards explaining this mystery, it has been suggested that commonly used gradient-based optimization algorithms enforce certain implicit regularization which effectively constrains the model capacity. (2) Benign overfitting: In certain scenarios, a model can perfectly fit noisily labeled training data, but still archives near-optimal test error at the same time, which is very different from the classical notion of overfitting. (3) Grokking: In certain scenarios, a model initially achieves perfect training accuracy but no generalization (i.e. no better than a random predictor), and upon further training, transitions to almost perfect generalization. Theoretically establishing these properties often involves making appropriate high-dimensional assumptions on the problem as well as a careful analysis of the training dynamics.

AC-SLODAR: measuring daytime normalized optical turbulence intensity distribution based on slope autocorrelation

ABSTRACT In solar observations, the vertical distribution of turbulence can be detected with a multidirectional Shack–Hartmann wavefront sensor, based on SLODAR or S-DIMM+. By expressing the measured cross-correlation as a linear combination of theoretical correlation functions for various height layers, the distribution of turbulence intensity can be obtained through fitting. The theoretical correlation functions in different heights are employed to describe the slope correlation and calculated based on the phase structure function corresponding to classical turbulence statistical theories such as Kolmogorov or von Karman. However, for turbulence that does not obey classical theory, this deviation of the statistical characteristics would result in the measurement accuracy degradation of seeing profiles. A method, so-called AutoCorrelation-SLODAR (AC-SLODAR), is proposed. The feasibility of transforming cross-correlation into autocorrelation is analysed theoretically. Then, the autocorrelation function is calculated based on the actual data to avoid the deviation introduced by the theoretical turbulence statistical model. Extracting statistical characteristics from data also simplifies measurements, without requiring the evaluation of whether the data conforms to a particular classical statistical model. AC-SLODAR was validated with simulation data generated by the open-source emulator SOAPY. The measured error was reduced by 10 per cent compared with SLODAR for the situation of turbulence model deviation. The performance of AC-SLODAR was further compared with those of SLODAR based on the actual data collected in 2016 and 2017 on the New Vacuum Solar Telescope. The corresponding results of AC-SLODAR are consistent with those of SLODAR using a pre-evaluated turbulent model.

Deep reinforcement learning applied to statistical arbitrage investment strategy on cryptomarket

In the complex and dynamic nature of financial markets where increasingly sophisticated investment strategies are required, deep reinforcement learning (DRL) has proven successful in generating real-time investment strategies, outperforming classical models. Alongside this, statistical arbitrage strategies exploit temporary market inefficiencies to generate returns. In this regard, a novel DRL-based arbitrage method has been developed.This paper presents a unified framework that combines classical statistical arbitrage theory with DRL framework to generate investment strategies. The framework addresses the challenges of identifying similar asset portfolios, extracting signals indicating temporary price deviations, and determining optimal trading rules given market conditions. The proposed methodology involves constructing arbitrage portfolios based on cointegration relationships, removing signals from price series and portfolios, and using a DRL agent to make optimal decisions within a fixed time horizon.The empirical analysis focuses on the cryptocurrency market, known for its volatility and risk. Results demonstrate that DRL agents can generate strategies with positive returns in out-of-sample periods 79.52% to 112.82% with no transaction cost, outperforming market benchmark Bitcoin 32.51% return, the best performing over the period Litecoin with 57.11% return and the worst performing Solana with a 35.70% loss. Moreover, these strategies effectively reduce risk, achieving higher risk-adjusted returns on individual assets. The strategies maintain positive returns when considering transaction costs, with the DRL agent outperforming the standard arbitrage strategy. The best-performing strategy is based on a Deep Q-Network (DQN) agent with a return of 18.39%, annualized volatility of 12.22%, and an annualized Sharpe ratio of 2.43. At the same time, Bitcoin holds an annualized volatility of 44.13% and a Sharpe ratio of 1.08.Their randomness and coherence are studied to verify the robustness of the agents’ decisions. The actions generated by the agents are not random but based on well-founded policies, which align with the obtained results.

Bild Conception of Scientific Theory Structuring in Classical and Quantum Physics: From Hertz and Boltzmann to Schrödinger and De Broglie.

We start with a methodological analysis of the notion of scientific theory and its interrelation with reality. This analysis is based on the works of Helmholtz, Hertz, Boltzmann, and Schrödinger (and reviews of D'Agostino). Following Helmholtz, Hertz established the "Bild conception" for scientific theories. Here, "Bild" ("picture") carries the meaning "model" (mathematical). The main aim of natural sciences is construction of the causal theoretical models (CTMs) of natural phenomena. Hertz claimed that a CTM cannot be designed solely on the basis of observational data; it typically contains hidden quantities. Experimental data can be described by an observational model (OM), often based on the price of acausality. CTM-OM interrelation can be tricky. Schrödinger used the Bild concept to create a CTM for quantum mechanics (QM), and QM was treated as OM. We follow him and suggest a special CTM for QM, so-called prequantum classical statistical field theory (PCSFT). QM can be considered as a PCSFT image, but not as straightforward as in Bell's model with hidden variables. The common interpretation of the violation of the Bell inequality is criticized from the perspective of the two-level structuring of scientific theories. Such critical analysis of von Neumann and Bell no-go theorems for hidden variables was performed already by De Broglie (and Lochak) in the 1970s. The Bild approach is applied to the two-level CTM-OM modeling of Brownian motion: the overdamped regime corresponds to OM. In classical mechanics, CTM=OM; on the one hand, this is very convenient; on the other hand, this exceptional coincidence blurred the general CTM-OM structuring of scientific theories. We briefly discuss ontic-epistemic structuring of scientific theories (Primas-Atmanspacher) and its relation to the Bild concept. Interestingly, Atmanspacher as well as Hertz claim that even classical physical theories should be presented on the basic of two-level structuring.

Quantum information criteria for model selection in quantum state estimation

Quantum state estimation (or state tomography) is an indispensable task in quantum information processing. Because full state tomography that determines all elements of the density matrix is computationally demanding, one usually takes the strategy of assuming a certain model of quantum states and identifying the model parameters. However, it is difficult to make a valid assumption given little prior knowledge on a quantum state of interest, and thus we need a reasonable model selection method for quantum state estimation. Actually, in the classical statistical estimation theory, several types of information criteria have been established and widely used in practice for appropriately choosing a classical statistical model. In this study, we propose quantum information criteria for evaluating the quality of the estimated quantum state in terms of the quantum relative entropy, which is a natural quantum analogue of the classical information criterion defined in terms of Kullback–Leibler divergence. In particular, we derive two quantum information criteria depending on the type of an estimator for the quantum relative entropy; one uses the log-likelihood and the other uses the classical shadow. The general role of information criteria is to predict the performance of an estimated model for unseen data, although it is a function of only sampled data; this generalization capability of the proposed quantum information criteria is evaluated in numerical simulations.

Analysis of genetic dominance in the UK Biobank.

Classical statistical genetics theory defines dominance as any deviation from a purely additive, or dosage, effect of a genotype on a trait, which is known as the dominance deviation. Dominance is well documented in plant and animal breeding. Outside of rare monogenic traits, however, evidence in humans is limited. We systematically examined common genetic variation across 1060 traits in a large population cohort (UK Biobank, N = 361,194 samples analyzed) for evidence of dominance effects. We then developed a computationally efficient method to rapidly assess the aggregate contribution of dominance deviations to heritability. Lastly, observing that dominance associations are inherently less correlated between sites at a genomic locus than their additive counterparts, we explored whether they may be leveraged to identify causal variants more confidently.

On Generalization Bounds for Deep Networks Based on Loss Surface Implicit Regularization

The classical statistical learning theory implies that fitting too many parameters leads to overfitting and poor performance. That modern deep neural networks generalize well despite a large number of parameters contradicts this finding and constitutes a major unsolved problem towards explaining the success of deep learning. While previous work focuses on the implicit regularization induced by stochastic gradient descent (SGD), we study here how the local geometry of the energy landscape around local minima affects the statistical properties of SGD with Gaussian gradient noise. We argue that under reasonable assumptions, the local geometry forces SGD to stay close to a low dimensional subspace and that this induces another form of implicit regularization and results in tighter bounds on the generalization error for deep neural networks. To derive generalization error bounds for neural networks, we first introduce a notion of stagnation sets around the local minima and impose a local essential convexity property of the population risk. Under these conditions, lower bounds for SGD to remain in these stagnation sets are derived. If stagnation occurs, we derive a bound on the generalization error of deep neural networks involving the spectral norms of the weight matrices but not the number of network parameters. Technically, our proofs are based on controlling the change of parameter values in the SGD iterates and local uniform convergence of the empirical loss functions based on the entropy of suitable neighborhoods around local minima.

On Maximally Mixed Equilibria of Two-Dimensional Perfect Fluids

The vorticity of a two-dimensional perfect (incompressible and inviscid) fluid is transported by its area preserving flow. Given an initial vorticity distribution $\omega_0$, predicting the long time behavior which can persist is an issue of fundamental importance. In the infinite time limit, some irreversible mixing of $\omega_0$ can occur. Since kinetic energy $\mathsf{E}$ is conserved, not all the mixed states are relevant and it is natural to consider only the ones with energy $\mathsf{E}_0$ corresponding to $\omega_0$. The set of said vorticity fields, denoted by $\overline{\mathcal{O}_{\omega_0}}^*\cap \{ {\mathsf E}= {\mathsf E}_0\}$, contains all the possible end states of the fluid motion. A. Shnirelman introduced the concept of maximally mixed states (any further mixing would necessarily change their energy), and proved they are perfect fluid equilibria. We offer a new perspective on this theory by showing that any minimizer of any strictly convex Casimir in $\overline{\mathcal{O}_{\omega_0}}^*\cap \{ {\mathsf E}= {\mathsf E}_0\}$ is maximally mixed, as well as discuss its relation to classical statistical hydrodynamics theories. Thus, (weak) convergence to equilibrium cannot be excluded solely on the grounds of vorticity transport and conservation of kinetic energy. On the other hand, on domains with symmetry (e.g. straight channel or annulus), we exploit all the conserved quantities and the characterizations of $\overline{\mathcal{O}_{\omega_0}}^*\cap \{ {\mathsf E}= {\mathsf E}_0\}$ to give examples of open sets of initial data which can be arbitrarily close to any shear or radial flow in $L^1$ of vorticity but do not weakly converge to them in the long time limit.

Probabilistic prediction for binary treatment choice: With focus on personalized medicine

This paper extends my research applying statistical decision theory to treatment choice with sample data, using maximum regret to evaluate the performance of statistical treatment rules. The specific new contribution is to study as-if optimization using estimates of illness probabilities in a class of medical decisions. Beyond its specifics, the paper sends a broad message. Statisticians and computer scientists have addressed conditional prediction for decision making in indirect ways, the former applying classical statistical theory and the latter measuring prediction accuracy in test samples. Neither approach is satisfactory. Statistical decision theory provides a coherent, generally applicable methodology.

Evaluating methods for Lasso selective inference in biomedical research: a comparative simulation study

BackgroundVariable selection for regression models plays a key role in the analysis of biomedical data. However, inference after selection is not covered by classical statistical frequentist theory, which assumes a fixed set of covariates in the model. This leads to over-optimistic selection and replicability issues.MethodsWe compared proposals for selective inference targeting the submodel parameters of the Lasso and its extension, the adaptive Lasso: sample splitting, selective inference conditional on the Lasso selection (SI), and universally valid post-selection inference (PoSI). We studied the properties of the proposed selective confidence intervals available via R software packages using a neutral simulation study inspired by real data commonly seen in biomedical studies. Furthermore, we present an exemplary application of these methods to a publicly available dataset to discuss their practical usability.ResultsFrequentist properties of selective confidence intervals by the SI method were generally acceptable, but the claimed selective coverage levels were not attained in all scenarios, in particular with the adaptive Lasso. The actual coverage of the extremely conservative PoSI method exceeded the nominal levels, and this method also required the greatest computational effort. Sample splitting achieved acceptable actual selective coverage levels, but the method is inefficient and leads to less accurate point estimates.The choice of inference method had a large impact on the resulting interval estimates, thereby necessitating that the user is acutely aware of the goal of inference in order to interpret and communicate the results.ConclusionsDespite violating nominal coverage levels in some scenarios, selective inference conditional on the Lasso selection is our recommended approach for most cases. If simplicity is strongly favoured over efficiency, then sample splitting is an alternative. If only few predictors undergo variable selection (i.e. up to 5) or the avoidance of false positive claims of significance is a concern, then the conservative approach of PoSI may be useful. For the adaptive Lasso, SI should be avoided and only PoSI and sample splitting are recommended. In summary, we find selective inference useful to assess the uncertainties in the importance of individual selected predictors for future applications.

Temperature-Dependent EXAFS Debye–Waller Factor of Distorted HCP Crystals

In this paper, an efficient theoretical model has been developed to calculate and analyze the extended X-ray absorption fine structure (EXAFS) Debye–Waller factor (DWF) of distorted hexagonal close-packed (HCP) crystals with non-ideal axial ratio c/a in the temperature dependence. The analysis procedure is performed by evaluating the influence of temperature on the amplitude reduction and phase shift of EXAFS oscillation that is expressed in terms of EXAFS DWF using EXAFS cumulants in the cumulant expansion approach. The calculation model of the temperature-dependent EXAFS cumulants is expanded from the correlated Einstein model using the classical statistical theory and the anharmonic effective potential that depends on non-ideal axial ratio c/a of distorted HCP crystals. The numerical results for crystalline zinc agree well with those obtained from other theoretical methods and experiments at various temperatures. The obtained results show the present theoretical model is necessary to analyze and evaluate the temperature-dependent experimental EXAFS signals of distorted HCP crystals under the effect of non-ideal axial ratio c/a.

Binary classification with classical instances and quantum labels

In classical statistical learning theory, one of the most well-studied problems is that of binary classification. The information-theoretic sample complexity of this task is tightly characterized by the Vapnik-Chervonenkis (VC) dimension. A quantum analog of this task, with training data given as a quantum state has also been intensely studied and is now known to have the same sample complexity as its classical counterpart. We propose a novel quantum version of the classical binary classification task by considering maps with classical input and quantum output and corresponding classical-quantum training data. We discuss learning strategies for the agnostic and for the realizable case and study their performance to obtain sample complexity upper bounds. Moreover, we provide sample complexity lower bounds which show that our upper bounds are essentially tight for pure output states. In particular, we see that the sample complexity is the same as in the classical binary classification task w.r.t. its dependence on accuracy, confidence and the VC-dimension.

A Permutation Test for High-Dimensional Covariance Matrix

In the case of “big p and small n”, classical statistical methods and theories are difficult to apply to high-dimensional data problems. This article considers testing the equality of two sample covariance matrices. When both the dimension p and the sample size n tend to infinity, a permutation method is proposed. The new test method eliminates the limitation of sample distribution and dimension. Numerical research shows that the proposed test method has good results in both normal and non-Gaussian distributions under high-dimensional data.

Simulation evaluation of small samples based on grey estimation and improved bootstrap

PurposeTo solve the problem of simulation evaluation with small samples, a fresh approach of grey estimation is presented based on classical statistical theory and grey system theory. The purpose of this paper is to make full use of the difference of data distribution and avoid the marginal data being ignored.Design/methodology/approachBased upon the grey distribution characteristics of small sample data, the definition about a new concept of grey relational similarity measure comes into being. At the same time, the concept of sample weight is proposed according to the grey relational similarity measure. Based on the new definition of grey weight, the grey point estimation and grey confidence interval are studied. Then the improved Bootstrap resampling is designed by uniform distribution and randomness as an important supplement of the grey estimation. In addition, the accuracy of grey bilateral and unilateral confidence intervals is introduced by using the new grey relational similarity measure approach.FindingsThe new small sample evaluation method can realize the effective expansion and enrichment of data and avoid the excessive concentration of data. This method is an organic fusion of grey estimation and improved Bootstrap method. Several examples are used to demonstrate the feasibility and validity of the proposed methods to illustrate the credibility of some simulation data, which has no need to know the probability distribution of small samples.Originality/valueThis research has completed the combination of grey estimation and improved Bootstrap, which makes more reasonable use of the value of different data than the unimproved method.

Distribution-Dependent Weighted Union Bound.

In this paper, we deal with the classical Statistical Learning Theory’s problem of bounding, with high probability, the true risk of a hypothesis h chosen from a set of m hypotheses. The Union Bound (UB) allows one to state that where is the empirical errors, if it is possible to prove that and , when h, , and are chosen before seeing the data such that and . If no a priori information is available and are set to , namely equally distributed. This approach gives poor results since, as a matter of fact, a learning procedure targets just particular hypotheses, namely hypotheses with small empirical error, disregarding the others. In this work we set the and in a distribution-dependent way increasing the probability of being chosen to function with small true risk. We will call this proposal Distribution-Dependent Weighted UB (DDWUB) and we will retrieve the sufficient conditions on the choice of and that state that DDWUB outperforms or, in the worst case, degenerates into UB. Furthermore, theoretical and numerical results will show the applicability, the validity, and the potentiality of DDWUB.

Probabilistic Prediction for Binary Treatment Choice: With Focus on Personalized Medicine

This paper extends my research applying statistical decision theory to treatment choice with sample data, using maximum regret to evaluate the performance of treatment rules. The specific new contribution is to study as-if optimization using estimates of illness probabilities in clinical choice between surveillance and aggressive treatment. Beyond its specifics, the paper sends a broad message. Statisticians and computer scientists have addressed conditional prediction for decision making in indirect ways, the former applying classical statistical theory and the latter measuring prediction accuracy in test samples. Neither approach is satisfactory. Statistical decision theory provides a coherent, generally applicable methodology.

Statistical Applications in Educational Measurement

Educational measurement assigns numbers to individuals based on observed data to represent individuals’ educational properties such as abilities, aptitudes, achievements, progress, and performance. The current review introduces a selection of statistical applications to educational measurement, ranging from classical statistical theory (e.g., Pearson correlation and the Mantel–Haenszel test) to more sophisticated models (e.g., latent variable, survival, and mixture modeling) and statistical and machine learning (e.g., high-dimensional modeling, deep and reinforcement learning). Three main subjects are discussed: evaluations for test validity, computer-based assessments, and psychometrics informing learning. Specific topics include item bias detection, high-dimensional latent variable modeling, computerized adaptive testing, response time and log data analysis, cognitive diagnostic models, and individualized learning.

Temperature dependence of EXAFS spectra of BCC crystals analyzed based on classical anharmonic correlated Einstein model

In this work, the temperature dependence of extended X-ray absorption fine structure (EXAFS) of the body-centered cubic crystals was investigated based on the anharmonic correlated Einstein model using the classical statistical theory. The oscillation of the anharmonic EXAFS spectra presented in terms of the cumulant expansion up to the fourth order. Here, the thermodynamic parameters of a system are derived from an anharmonic effective potential that has taken into account the influence of all nearest neighbors of absorbing and backscattering atoms. Analytical expressions of the first four EXAFS cumulants are obtained in simple forms of temperature or parallel mean-square relative displacement. The numerical results for crystalline molybdenum in the temperature range from 0 to 900 K are found to be in good agreement with those obtained using the other theoretical methods and experiments at various temperatures. The analytical results for the contributions of the cumulants to the amplitude reduction and phase shift of the EXAFS oscillation discover the role and meaning of high-order cumulants in analyzing the temperature dependence of the EXAFS spectra.

Gaussian determinantal processes: A new model for directionality in data

Determinantal point processes (DPPs) have recently become popular tools for modeling the phenomenon of negative dependence, or repulsion, in data. However, our understanding of an analogue of a classical parametric statistical theory is rather limited for this class of models. In this work, we investigate a parametric family of Gaussian DPPs with a clearly interpretable effect of parametric modulation on the observed points. We show that parameter modulation impacts the observed points by introducing directionality in their repulsion structure, and the principal directions correspond to the directions of maximal (i.e., the most long-ranged) dependency. This model readily yields a viable alternative to principal component analysis (PCA) as a dimension reduction tool that favors directions along which the data are most spread out. This methodological contribution is complemented by a statistical analysis of a spiked model similar to that employed for covariance matrices as a framework to study PCA. These theoretical investigations unveil intriguing questions for further examination in random matrix theory, stochastic geometry, and related topics.

Multivariate statistics and its applications

Multivariate statistics and its applications have received more and more attention since 1950s.Multivariate statistical research in China was initiated by Pao-Lu Hsu duringthe end of 1930s and the beginning of 1940s in the so-called “Southwest United University”.Modern big data analysis makes classical multivariate statistical theory unable to accurately andeffectively solve practical problems. The theory of generalized multivariate statistics has beendeveloping very fast since 1970s. This paper aims to introduce the summary contributions that Chinesescholars have made in the development of generalized multivariate statistics and its applications inseveral aspects: (1) multivariate statistics and generalized multivariate statistics; (2) general symmetricmultivariate distributions; and (3) growth curve modeling and miscellaneous directions.Generalized multivariate statistics is an extension to the traditional statistics under the normalassumption. It aims to generalize the traditional statistical methodologies like parametric estimation,hypothesis testing, and modeling to a much wider family of multivariate distributionsthat are called elliptically contoured distributions (ECDs). General symmetric multivariate distributionsform an even wider class of multivariate probability distributions that includes the ECDs as its special case.Growth curve modeling (GCM) includes statistical methods that allow for consideration of inter-individualvariability in intra-individual patterns of chance over time. Outlier detection and identification ofinfluential observations are important topics in the area of GCM.