Finite Sample Size Research Articles

Value-of-Information Analysis for External Validation of Risk Prediction Models.

A previously developed risk prediction model needs to be validated before being used in a new population. The finite size of the validation sample entails that there is uncertainty around model performance. We apply value-of-information (VoI) methodology to quantify the consequence of uncertainty in terms of net benefit (NB). We define the expected value of perfect information (EVPI) for model validation as the expected loss in NB due to not confidently knowing which of the alternative decisions confers the highest NB. We propose bootstrap-based and asymptotic methods for EVPI computations and conduct simulation studies to compare their performance. In a case study, we use the non-US subsets of a clinical trial as the development sample for predicting mortality after myocardial infarction and calculate the validation EVPI for the US subsample. The computation methods generated similar EVPI values in simulation studies. EVPI generally declined with larger samples. In the case study, at the prespecified threshold of 0.02, the best decision with current information would be to use the model, with an incremental NB of 0.0020 over treating all. At this threshold, the EVPI was 0.0005 (relative EVPI = 25%). When scaled to the annual number of heart attacks in the US, the expected NB loss due to uncertainty was equal to 400 true positives or 19,600 false positives, indicating the value of further model validation. VoI methods can be applied to the NB calculated during external validation of clinical prediction models. While uncertainty does not directly affect the clinical implications of NB findings, validation EVPI provides an objective perspective to the need for further validation and can be reported alongside NB in external validation studies. External validation is a critical step when transporting a risk prediction model to a new setting, but the finite size of the validation sample creates uncertainty about the performance of the model.In decision theory, such uncertainty is associated with loss of net benefit because it can prevent one from identifying whether the use of the model is beneficial over alternative strategies.We define the expected value of perfect information for external validation as the expected loss in net benefit by not confidently knowing if the use of the model is net beneficial.The adoption of a model for a new population should be based on its expected net benefit; independently, value-of-information methods can be used to decide whether further validation studies are warranted.

Ensuring valid inference for Cox hazard ratios after variable selection.

The problem of how to best select variables for confounding adjustment forms one of the key challenges in the evaluation of exposure effects in observational studies, and has been the subject of vigorous recent activity in causal inference. A major drawback of routine procedures is that there is no finite sample size at which they are guaranteed to deliver exposure effect estimators and associated confidence intervals with adequate performance. In this work, we will consider this problem when inferring conditional causal hazard ratios from observational studies under the assumption of no unmeasured confounding. The major complication that we face with survival data is that the key confounding variables may not be those that explain the censoring mechanism. In this paper, we overcome this problem using a novel and simple procedure that can be implemented using off-the-shelf software for penalized Cox regression. In particular, we will propose tests of the null hypothesis that the exposure has no effect on the considered survival endpoint, which are uniformly valid under standard sparsity conditions. Simulation results show that the proposed methods yield valid inferences even when covariates arehigh-dimensional.

Data weighting: An iterative process linking surveys, data synthesis, and population models to evaluate mis-specification

Integrated stock assessments specify a distribution for multiple data types, and these distributions control the relative leverage assigned to each datum. A decade of research has demonstrated that (1) proper data weighting is necessary to avoid bias resulting from overweighting noisy age- and length-composition data; (2) sampling data can be pre-processed to estimate the likely sampling variance for composition data; and (3) using random effects to estimate time-varying parameters can improve the fit to data while also changing statistical leverage, and thereby serve a similar role to reweighting data. However, there are also unresolved questions including: (A) Is it more appropriate to model age and length data as proportions-at-age and as an index for the total, or as a series of indices-at-age? (B) Are correlated residuals appropriately addressed via data weighting or do they require additional model changes (i.e., time-varying parameters)? (C) How to efficiently communicate information about sampling imprecision and model errors between sampling and stock-assessment teams? (D) How does model-based expansion of sampling data affect data weighting? and (E) How to address alternative hypotheses about factors driving poor fit to data? Here, we argue that stock assessment errors can be classified using four categories: sampling bias (e.g., changes in survey coverage), sampling imprecision (e.g., finite sample sizes), assessment model bias (e.g., incorrect demographic assumptions) and assessment model imprecision (e.g., random effects). This categorization has several implications with resulting practical recommendations. For example, we define Percent Excess Variance (PEV) from the ratio of input sample size (the measured variance of sampling imprecision) and effective sample sizes (the variance of assessment-model residuals). We propose calculating PEV as standardized diagnostic measuring the net effect of survey bias and assessment model bias and imprecision. We demonstrate PEV in a simulation experiment fitted using the Woods Hole Assessment Model (WHAM) conditioned upon Gulf of Alaska walleye pollock, where unacknowledged variation in fishery selectivity results in a PEV of 77 % and this is eliminated when correctly specifying a time-varying estimation model. We also argue that model-based expansion of data inputs using auxiliary information can mitigate sampling bias, while also measuring sampling imprecision for spatially unrepresentative surveys. Similarly, including random effects can mitigate model bias while increasing model imprecision when the demographic model has little explanatory power. Finally, we observe that down-weighting compositional data for a given fleet fails to propagate information about model residuals when interpreting abundance indices or reference points for that same fleet. When PEV is large for important fleets, we therefore encourage focused research to explain the sources of these errors rather than simply downweighting without propagating information about residuals. However, we acknowledge a continuing role for automated data weighting for less important fleets, although we recommend explicit hypotheses about potential sources of errors in those cases.

A New Test on Asset Return Predictability with Structural Breaks

This article considers predictive regressions in which a structural break is allowed on an unknown date. We establish novel testing procedures for asset return predictability using empirical likelihood (EL) methods based on weighted score equations. The theoretical results are useful in practice because our unified framework does not require distinguishing whether the predictor variables are stationary or non-stationary. Monte Carlo simulation studies show that the EL-based tests perform well in terms of size and power in finite samples. Finally, as an empirical analysis, we test the predictability of the monthly S&P 500 value-weighted log excess return using various predictor variables.

Reduced bias estimation of the shape parameter of the log-logistic distribution

In the literature, the log-logistic distribution is commonly presented with two parameters: one that governs the shape of the model, and the other that governs its scale. However, to make this model more suitable for data analysis, an additional location parameter can be added, resulting in the three-parameter or shifted log-logistic model. In this paper, we introduce a new estimator for the shape parameter of a three-parameter log-logistic distribution that reduces bias. We also derive various properties of the proposed estimator. Additionally, a simulation study and an application example to a real data set are conducted to examine the efficiency for finite sample sizes. The theoretical and simulated results confirm that our proposed estimation method performs significantly better than other estimation methods found in the literature.

Only Classical Parameterised States have Optimal Measurements under Least Squares Loss

Measurements of quantum states form a key component in quantum-information processing. It is therefore an important task to compare measurements and furthermore decide if a measurement strategy is optimal. Entropic quantities, such as the quantum Fisher information, capture asymptotic optimality but not optimality with finite resources. We introduce a framework that allows one to conclusively establish if a measurement is optimal in the non-asymptotic regime. Our method relies on the fundamental property of expected errors of estimators, known as risk, and it does not involve optimisation over entropic quantities. The framework applies to finite sample sizes and lack of prior knowledge, as well as to the asymptotic and Bayesian settings. We prove a no-go theorem that shows that only classical states admit optimal measurements under the most common choice of error measurement: least squares. We further consider the less restrictive notion of an approximately optimal measurement and give sufficient conditions for such measurements to exist. Finally, we generalise the notion of when an estimator is inadmissible (i.e. strictly worse than an alternative), and provide two sufficient conditions for a measurement to be inadmissible.

Kernel regression for cause-specific hazard models with nonparametric covariate functions

We study the local kernel pseudo-partial likelihood approach for the cause-specific hazard model with nonparametric covariate functions. The derivative of the covariate function is estimated first, and the estimator of the nonparametric covariate function is then derived by integrating the derivative estimator. The consistency and pointwise asymptotic normality of the local kernel estimator for the interested failure types are obtained. Moreover, numerical studies show that the proposed kernel estimator performs well under a finite sample size. And we compare the local kernel estimator with the regression B-splines estimator. We also apply the proposed method to analyse the kidney and renal pelvis cancer data with composite endpoints.

A robust estimator of mutual information for deep learning interpretability

We develop the use of mutual information (MI), a well-established metric in information theory, to interpret the inner workings of deep learning (DL) models. To accurately estimate MI from a finite number of samples, we present GMM-MI (pronounced ‘Jimmie’), an algorithm based on Gaussian mixture models that can be applied to both discrete and continuous settings. GMM-MI is computationally efficient, robust to the choice of hyperparameters and provides the uncertainty on the MI estimate due to the finite sample size. We extensively validate GMM-MI on toy data for which the ground truth MI is known, comparing its performance against established MI estimators. We then demonstrate the use of our MI estimator in the context of representation learning, working with synthetic data and physical datasets describing highly non-linear processes. We train DL models to encode high-dimensional data within a meaningful compressed (latent) representation, and use GMM-MI to quantify both the level of disentanglement between the latent variables, and their association with relevant physical quantities, thus unlocking the interpretability of the latent representation. We make GMM-MI publicly available in this GitHub repository.

Localized spin-wave modes and microwave absorption in random-anisotropy ferromagnets

The theory of localized spin-wave excitations in random-anisotropy magnets has been developed. Starting with a pure Heisenberg ferromagnet, we study the evolution of standing spin waves in a finite-size sample towards localized modes on increasing the strength of random anisotropy. Profiles of the localized modes and their phases are analyzed and visualized in a 2D sample. Localization length is obtained by several methods and its dependence on random anisotropy is computed. The connection between the localization of spin excitations and the broadband nature of the absorption of microwave power by random-anisotropy magnets is elucidated.

ПРОБЛЕМА ОПТИМАЛЬНОСТІ АДАПТИВНИХ РАДІОТЕХНІЧНИХ СИСТЕМ В СИТУАЦІЇ З НЕКЛАСИФІКОВАНОЮ НАВЧАЛЬНОЮ ВИБІРКОЮ СПОСТЕРЕЖЕНЬ

Statistical synthesis of adaptive systems for spatiotemporal signal processing in most radio engineering problems occurs under the condition of parametric a priori uncertainty. The prospective direction of overcoming parametric a priori uncertainty is affiliated with the prior learning procedure. As a result of the training, sufficient statistics of likelihood ratios are formed, which are used in decision-making. The asymptotic optimality of adaptive systems with alternative benchmarks, one of which is peculiar to the class of radio structures with an adaptive antenna array and the other with an adaptive interference compensator, is studied subjectively with unclassified training sampling. The optimality of the system is determined by the criterion of maximum signal-to-interference ratio. The energy parameter representing the signal-to-interference ratio at the output of adaptive radio systems with a generalized reference was chosen as an indicator of optimality. As a result of the research, analytical expressions for the estimation and comparative analysis of energy losses at the output of adaptive systems with different standards under conditions of classified and unclassified learning were obtained. The invariance of the system with an adaptive antenna array to any type of training with preservation of the asymptotic properties of the radio system in the situation with a finite size of the training sample is proved. This feature is considered as a basis for factorization of the transfer function of the adaptive system. This makes it possible to form an optimal hierarchical algorithm for compensation of combined passive and active interference, as well as active interference with an arbitrary spatial power spectrum. The obligatory classification of a training sample of observations for radio engineering systems with an adaptive interference compensator under conditions of signal-interference a priori uncertainty is substantiated. Keywords: asymptotic optimality, adaptive system, alternative benchmark, unclassified training sample, correlation matrix of observations, energy index.

A new spin on color quantization

We address the problem of image color quantization using a maximum entropy based approach. Focusing on pixel mapping we argue that adding thermal noise to the system yields better visual impressions than that obtained from a simple energy minimization. To quantify this observation, we introduce the coarse-grained quantization error, and seek the optimal temperature which minimizes this new observable. By comparing images with different structural properties, we show that the optimal temperature is a good proxy for complexity at different scales. Noting that the convoluted error is a key observable, we directly minimize it using a Monte Carlo algorithm to generate a new series of quantized images. Adopting an original approach based on the informativity of finite size samples, we are able to determine the optimal convolution parameter leading to the best visuals. Finally, we test the robustness of our method against changes in image type, color palette and convolution kernel.

Asymptotic Behavior of Temporal Aggregation in Mixed‐Frequency Datasets

AbstractHere, we present an unexplored issue regarding temporal aggregation. When a model contains frequency‐dependent coefficients, such as a distinct long‐ and short‐term coefficient, temporal aggregation leads to inconsistent least squares estimates. Because the sub‐sampled variable's spectrum is equal to its folded original spectrum, the low‐frequency variable may exhibit a mixture of distinct linear relations for a given frequency. We propose a new method to disentangle the frequencies superposition based on band spectrum regression, thus avoiding the inconsistency problem. As a result, we can test for the presence of frequency‐dependent coefficients. We use stationary and non‐stationary linear semi‐parametric models to demonstrate our findings. Our Monte Carlo simulations show good finite sample size and power properties. Finally, our empirical study rejects the presence of a single coefficient for all frequencies between quarterly GDP and monthly US indicators.

Acoustic graphyne: A second-order real Chern topological insulator

Graphyne has recently attracted much attention since it is an important derivative of graphene with unique topological properties. Although graphyne is not a conventional topological insulator because of its weak spin–orbit coupling, it is a real Chern topological insulator with the higher-order topology. However, it lacks a realistic model. Here, we propose a schedule to realize acoustic graphyne. By introducing negative coupling to simulate the carbon–carbon triple bond, we realize the transition from trivial to higher-order topological phases, characterized by real Chern numbers. These topologically protected corner states are achieved in a finite-size sample, and the condition for their existence is discussed. Our research extends the concept of real Chern insulators and provides a platform for studying the topological properties of graphene-like structural compounds.

Bootstrapping tests for breaks in mean or variance based on U-statistics

Bootstrapping tests are implemented for the tests U by Schmidt [Detecting changes in the trend function of heteroscedastic time series; 2021. Preprint: arXiv:2108.09206 [math.ST]] for mean break and Schmidt et al. [An asymptotic test for constancy of the variance under short-range dependence. Ann Stat. 2021;49:3460–3481.] for variance break based on U-statistics. The tests U have good powers against epidemic breaks that are common in practice. The test U for variance break is proved to have the nice property of consistency against a general class of alternatives. However, the tests U have non-ignorable finite sample size distortion under serial correlation and/or conditional heteroscedasticity. Our implementation based on autoregressive residual bootstrapping and moving block bootstrapping are shown to remedy the size distortion problems of U for mean break and for variance break, respectively, in a Monte-Carlo experiment. The experiment also demonstrates the power advantages of bootstrapping tests over the original tests and other standard break tests against epidemic breaks, which, however, are accompanied by disadvantages against simple single breaks. The proposed bootstrapping tests are well illustrated by break analyses of means and variances of the log-return and realized the volatility of the US S&P 500.

Taking stock of long-horizon predictability tests: Are factor returns predictable?

This study provides a critical assessment of long-horizon return predictability tests using highly persistent regressors. We show that the commonly used statistics are typically oversized, leading to spurious inference. Instead, we propose a Wald statistic, which accommodates multiple predictors of (unknown) arbitrary persistence degree within the I(0)-I(1) range. The test statistic, based on an adaptation of the IVX procedure to a long-horizon regression framework, is shown to have a standard chi-squared asymptotic distribution (regardless of the stochastic properties of the regressors used as predictors) and to exhibit excellent finite-sample size and power properties. Employing this test statistic, we find evidence of predictability for “old” and “new” pricing factors with monthly returns, but this becomes weaker as the predictive horizon increases. The predictability evidence substantially weakens with annual data. Overall, we question the incremental value of using long-horizon predictive regressions.

Jackknife estimation of a cluster-sample IV regression model with many weak instruments

This paper proposes new jackknife IV estimators that are robust to the effects of many weak instruments and error heteroskedasticity in a cluster sample setting with cluster-specific effects and possibly many included exogenous regressors. The estimators that we propose are designed to properly partial out the cluster-specific effects and included exogenous regressors while preserving the re-centering property of the jackknife methodology. To the best of our knowledge, our proposed procedures provide the first consistent estimators under many weak instrument asymptotics in the setting considered. We also present results on the asymptotic normality of our estimators and show that t-statistics based on said estimators are asymptotically normal under the null and consistent under fixed alternatives. Monte Carlo results show that our t-statistics perform better in controlling size in finite samples than those based on alternative jackknife IV procedures previously introduced in the literature.

Breaks in term structures: Evidence from the oil futures markets

AbstractWe propose a new functional change point detection procedure, motivated by recent models for commodity futures term structure. We investigate our procedure's properties under the null hypothesis of no change and the alternative. Monte Carlo simulations reveal a reasonable power property in finite sample sizes although size distortion persists. An empirical analysis of oil futures markets identifies two change points near the 2008 financial crisis and 2020 crude oil negative territory. Regression models show that the price behaviour, in general, is exposed to the spot market index and exchange rate from 2007 to 2009. The main drivers of the price term structure are attributed to the trading activities of speculators and financial index innovations between 2017 and 2022.

Accurate and fast small p-value estimation for permutation tests in high-throughput genomic data analysis with the cross-entropy method.

Permutation tests are widely used for statistical hypothesis testing when the sampling distribution of the test statistic under the null hypothesis is analytically intractable or unreliable due to finite sample sizes. One critical challenge in the application of permutation tests in genomic studies is that an enormous number of permutations are often needed to obtain reliable estimates of very small p-values, leading to intensive computational effort. To address this issue, we develop algorithms for the accurate and efficient estimation of small p-values in permutation tests for paired and independent two-group genomic data, and our approaches leverage a novel framework for parameterizing the permutation sample spaces of those two types of data respectively using the Bernoulli and conditional Bernoulli distributions, combined with the cross-entropy method. The performance of our proposed algorithms is demonstrated through the application to two simulated datasets and two real-world gene expression datasets generated by microarray and RNA-Seq technologies and comparisons to existing methods such as crude permutations and SAMC, and the results show that our approaches can achieve orders of magnitude of computational efficiency gains in estimating small p-values. Our approaches offer promising solutions for the improvement of computational efficiencies of existing permutation test procedures and the development of new testing methods using permutations in genomic data analysis.

On portmanteau-type tests for nonlinear multivariate time series

A general framework to devise portmanteau-type test statistics for a general class of multivariate nonlinear time series models with vector martingale difference errors is formulated. Based on this framework a suite of individual and mixed multivariate test statistics is considered. Two applications are developed: single- and multiple-lag test statistics. In each case, the resulting portmanteau test statistic is based on multivariate residuals and multivariate squared residuals. Moreover, single- and multiple-lag mixed multivariate portmanteau-type tests are introduced. These test statistics are designed to detect different forms of inadequacies in the model residuals jointly. All proposed tests take uncertainty due to model estimation properly into account. The asymptotic null distribution of each test statistic follows from the asymptotic distribution of the general portmanteau-type test statistic in a natural way. Some considerations are given to the empirical size and power of six test statistics via a simulation study. All tests have satisfactory size and power properties in finite samples. To demonstrate their practical use, the proposed test statistics are applied to the residuals of a vector bivariate nonlinear threshold model fitted to U.S. interest rates.

Dirac fermion spectrum of the fractional quantum Hall states

Applying a unified approach, we study the integer quantum Hall effect (IQHE) and fractional quantum Hall effect (FQHE) in the Hofstadter model with short range interactions between fermions. An effective field, that takes into account the interaction between fermions, is determined by both amplitude and phase. Its amplitude is proportional to the interaction strength, the phase corresponds to the minimum energy. In fact, the problem is reduced to the Harper equation with two different scales: the first is a magnetic scale with the cell size corresponding to a unit quantum magnetic flux, the second scale determines the inhomogeneity of the effective field, forms the steady fine structure of the Hofstadter spectrum and leads to the realization of fractional quantum Hall states. In a sample of finite size with open boundary conditions, the fine structure of the Hofstadter spectrum consists of the Dirac branches of the fermion excitations and includes the fine structure of the edge chiral modes. The Chern numbers of the topological Hofstadter bands are conserved during the formation of their fine structure. The edge modes are formed into the Hofstadter bands. They connect the nearest-neighbor subbands and determine the conductance for the fractional filling.