Statistical Problems Research Articles

Anytime valid and asymptotically efficient inference driven by predictive recursion

Abstract Distinguishing two candidate models is a fundamental and practically important statistical problem. Error rate control is crucial to the testing logic but, in complex nonparametric settings, can be difficult to achieve, especially when the stopping rule that determines the data collection process is not available. This paper proposes an e-process construction based on the predictive recursion algorithm originally designed to recursively fit nonparametric mixture models. The resulting predictive recursion process affords anytime valid inference and is asymptotically efficient in the sense that its growth rate is first-order optimal relative to the predictive recursion’s mixture model.

Estimation of Contact Time Among Animals from Telemetry Data

Continuous processes in most applications are measured discretely with error. This complicates the task of detecting intersections and the number of intersections between two continuous processes (i.e., when the processes have the same value). Intersections of continuous processes are scientifically important, but challenging to estimate from data. For example, in the field of animal ecology, intersections of the paths of moving animals tracked with satellite technologies can be used to understand disease transmission. We illustrate how to quantify contact between animals using telemetry data (i.e., the recorded locations of an animal over time). We introduce our method to quantify contact time with accessible concepts from introductory stochastic process literature, such as Brownian motion. Then, we provide two data examples using white-tailed deer (Odocoileus virginianus) and mule deer (Odocoileus hemionus) telemetry data in a region with high prevalence of chronic wasting disease. Our work provides a needed connection between existing model-based literature for animal movement and rule-based literature for animal interaction. Further, our work illustrates a unique statistical problem receiving minimal attention with broad applicability in human and livestock tracking. Supplementary materials for this article are available online.

Testing the equality of distributions using integrated maximum mean discrepancy

Comparing and testing for the homogeneity of two independent random samples is a fundamental statistical problem with many applications across various fields. However, existing methods may not be effective when the data is complex or high-dimensional. We propose a new method that integrates the maximum mean discrepancy (MMD) with a Gaussian kernel over all one-dimensional projections of the data. We derive the closed-form expression of the integrated MMD and prove its validity as a distributional similarity metric. We estimate the integrated MMD with the U-statistic theory and study its asymptotic behaviors under the null and two kinds of alternative hypotheses. We demonstrate that our method has the benefits of the MMD, and outperforms existing methods on both synthetic and real datasets, especially when the data is complex and high-dimensional.

Condition Monitoring Using a Latent Space of Variational Autoencoder Trained Only on a Healthy Machine.

Machine learning generative models have opened up a new perspective for automated machine diagnostics. These methods improve decision-making by extracting features, classifying, and creating new observations using deep neural networks. Generative modeling aims to determine the joint distribution of input data. This contrasts traditional methods used in diagnostics based on discriminative models and the conditional probability distribution of the target variable at known feature values. In the variational autoencoder (VAE) algorithms trained by the authors, the parameters of diagnostic features are random variables, the distributions of which can be approximated based on data, and the identification of probability distributions is based on variational inference. Variational inference is a tool that deals with difficult statistical problems and is usually faster than classical methods. VAEs can detect anomalies, predict failures, and optimize processes. This paper proposes an unsupervised approach to fault diagnosis using only healthy data with automatic feature extraction from the continuous probabilistic latent subspace of the VAE encoder and reduction in PCA or t-SNE. The solution, verified in the example of simulation data, is a response to a common problem related to the lack or difficulty of obtaining marked data in defected states of devices and mechanical structures.

Probabilistic analysis of tumor growth inhibition models to Support trial design

A large enough sample size of patients is required to statistically show that one treatment is better than another. However, too large a sample size is expensive and can also result in findings that are statistically significant, but not clinically relevant. How sample sizes should be chosen is a well-studied problem in classical statistics and analytical expressions can be derived from the appropriate test statistic. However, these expressions require information regarding the efficacy of the treatment, which may not be available, particularly for newly developed drugs. Tumor growth inhibition (TGI) models are frequently used to quantify the efficacy of newly developed anticancer drugs. In these models, the tumor growth dynamics are commonly described by a set of ordinary differential equations containing parameters that must be estimated using experimental data.One widely used endpoint in clinical trials is the proportion of patients in different response categories determined using the Response Evaluation Criteria In Solid Tumors (RECIST) framework. From the TGI model, we derive analytical expressions for the probability of patient response to combination therapy. The probabilistic expressions are used together with classical statistics to derive a parametric model for the sample size required to achieve a certain significance level and test power when comparing two treatments.Furthermore, the probabilistic expressions are used to generalize the Tumor Static Exposure concept to be more suitable for predicting clinical response. The derivatives of the probabilistic expressions are used to derive two additional expressions characterizing the exposure and its sensitivity. Finally, our results are illustrated using parameters obtained from calibrating the model to preclinical data.

Observing Quantum Measurement Collapse as a Learnability Phase Transition

During a quantum measurement, superpositions of states with different observable properties probabilistically collapse into one with a sharp value of the measured observable. In macroscopic quantum systems, this collapse arises via a continuous measurement-induced phase transition (MIPT) at a critical value of the strength of interaction with the measurement apparatus. MIPTs lie outside established paradigms for equilibrium or nonequilibrium critical phenomena and delineate distinct, stable dynamical and computational phases of matter. Quantum computers enable programmable simulation of the interaction of a measurement apparatus with a dynamical quantum system, to explore MIPT phenomena over a range of system sizes while retaining quantum coherence. Yet, existing experimental protocols rely on fundamentally nonscalable postselection techniques or direct classical simulation of quantum circuits. Here, we report the scalable observation of finite-size scaling evidence for an observable-sharpening MIPT in monitored quantum circuits in a chain of Yb+171 ions in Quantinuum’s H1-1 trapped-ion quantum processor. By leveraging an equivalent description as a statistical physics problem, we implement scalable classical algorithms to infer the value of the measured observable from a single experimental shot. This technique enables a truly scalable protocol to observe observable-sharpening MIPTs in generic classes of circuits that cannot be directly classically simulated and also provides enhanced means to detect and suppress errors in the quantum simulation. Published by the American Physical Society 2024

A flexible and efficient algorithm for high dimensional support vector regression

In high dimensional statistical learning, variable selection and handling highly correlated phenomena are two crucial topics. Elastic-net regularization can automatically perform variable selection and tends to either simultaneously select or remove highly correlated variables. Consequently, it has been widely applied in machine learning. In this paper, we incorporate elastic-net regularization into the support vector regression model, introducing the Elastic-net Support Vector Regression (En-SVR) model. Due to the inclusion of elastic-net regularization, the En-SVR model possesses the capability of variable selection, addressing high dimensional and highly correlated statistical learning problems. However, the optimization problem for the En-SVR model is rather complex, and common methods for solving the En-SVR model are challenging. Nevertheless, we observe that the optimization problem for the En-SVR model can be reformulated as a convex optimization problem where the objective function is separable into multiple blocks and connected by an inequality constraint. Therefore, we employ a novel and efficient Alternating Direction Method of Multipliers (ADMM) algorithm to solve the En-SVR model, and provide a complexity analysis as well as convergence analysis for the algorithm. Furthermore, extensive numerical experiments validate the outstanding performance of the En-SVR model in high dimensional statistical learning and the efficiency of this novel ADMM algorithm.

Improving Power Spectrum Estimation Using Multitapering: Efficient Asteroseismic Analyses for Understanding Stars, the Milky Way, and Beyond

Asteroseismic time series data have imprints of stellar oscillation modes, whose detection and characterization through time series analysis allows us to probe stellar interior physics. Such analyses usually occur in the Fourier domain by computing the Lomb–Scargle (LS) periodogram, an estimator of the power spectrum underlying unevenly sampled time series data. However, the LS periodogram suffers from the statistical problems of (1) inconsistency (or noise) and (2) bias due to high spectral leakage. Here, we develop a multitaper power spectrum estimator using the nonuniform fast Fourier transform (mtNUFFT) to tackle the inconsistency and bias problems of the LS periodogram. Using a simulated light curve, we show that the mtNUFFT power spectrum estimate of solar-like oscillations has lower variance and bias than the LS estimate. We also apply our method to the Kepler-91 red giant, and combine it with PBjam peakbagging to obtain mode parameters and a derived age estimate of 3.97 ± 0.52 Gyr. PBjam allows the improvement of age precision relative to the 4.27 ± 0.75 Gyr APOKASC-2 (uncorrected) estimate, whereas partnering mtNUFFT with PBjam speeds up peakbagging thrice as much as LS. This increase in efficiency has promising implications for Galactic archaeology, in addition to stellar structure and evolution studies. Our new method generally applies to time-domain astronomy and is implemented in the public Python package tapify, available at https://github.com/aaryapatil/tapify.

Exploring the Effectiveness of Advanced Chatbots in Educational Settings: A Mixed-Methods Study in Statistics

The Generative Pre-trained Transformer (GPT) is a highly advanced natural language processing model. This model can generate conversation-style responses to user input. The rapid rise of GPT has transformed academic domains, with studies exploring the potential of chatbots in education. This research investigates the effectiveness of ChatGPT 3.5, ChatGPT 4.0 by OpenAI, and Chatbot Bing by Microsoft in solving statistical exam-type problems in the educational setting. In addition to quantifying the errors made by these chatbots, this study seeks to understand the causes of these errors to provide recommendations. A mixed-methods approach was employed to achieve this goal, including quantitative and qualitative analyses (Grounded Theory with semi-structured interviews). The quantitative stage involves statistical problem-solving exercises for undergraduate engineering students, revealing error rates based on the reason for the error, statistical fields, sub-statistics fields, and exercise types. The quantitative analysis provided crucial information necessary to proceed with the qualitative study. The qualitative stage employs semi-structured interviews with selected chatbots; this includes confrontation between them that generates agreement, disagreement, and differing viewpoints. On some occasions, chatbots tend to maintain rigid positions, lacking the ability to adapt or acknowledge errors. This inflexibility may affect their effectiveness. The findings contribute to understanding the integration of AI tools in education, offering insights for future implementations and emphasizing the need for critical evaluation and responsible use.

Measuring environmental inefficiency through machine learning: An approach based on efficiency analysis trees and by-production technology

The main objective of this study is to introduce machine learning-type extensions for the measurement of environmental inefficiency based on regression trees under shape constraints. The new methods developed are implemented using a by-production approach that distinguishes two technologies, one related to the generation of pollution and the other to the production of good outputs. In particular, we define two alternative approaches to measuring environmental inefficiency: by-production Efficiency Analysis Trees (by-production EAT) and by-production Convexified Efficiency Analysis Trees (by-production CEAT). The main advantage of the methods developed is that they do not suffer from the typical statistical problem of overfitting connected to Free Disposal Hull (FDH) and Data Envelopment Analysis (DEA). The performance of the new models is evaluated through a simulation study which shows that the new approaches outperform FDH and DEA in terms of mean squared error and bias. We also illustrate the practical usefulness of the new techniques through empirical application to 43 developing and developed countries over a fifteen-year period - from 2000 to 2014. Our empirical findings using real data clearly indicate the higher discriminating power of the by-production EAT and CEAT models as compared respectively to FDH and DEA.

Subsampling Suffices for Adaptive Data Analysis

Ensuring that analyses performed on a dataset are representative of the entire population is one of the central problems in statistics. Most classical techniques assume that the dataset is independent of the analyst’s query and break down in the common setting where a dataset is reused for multiple, adaptively chosen, queries. This problem of adaptive data analysis was formalized in the seminal works of Dwork et al. (STOC, 2015) and Hardt and Ullman (FOCS, 2014). We identify a remarkably simple set of assumptions under which the queries will continue to be representative even when chosen adaptively: The only requirements are that each query takes as input a random subsample and outputs few bits. This result shows that the noise inherent in subsampling is sufficient to guarantee that query responses generalize. The simplicity of this subsampling-based framework allows it to model a variety of real-world scenarios not covered by prior work. In addition to its simplicity, we demonstrate the utility of this framework by designing mechanisms for two foundational tasks, statistical queries and median finding. In particular, our mechanism for answering the broadly applicable class of statistical queries is both extremely simple and state of the art in many parameter regimes.

Decision-Based Learning as an Approach for Teaching Statistics in a Peruvian University

Many students find learning statistics difficult, especially novice learners. Instructors also struggle to teach statistics due to many challenges, such as students’ misconceptions, lack of math skills, negative attitudes, and deficient instructional methods. A new pedagogical approach called Decision-Based Learning (DBL) has been developed to help instructors and students; however there are limited studies about teachers’ and students’ experiences with this method, particularly in international settings. For this project, a group of Peruvian university professors were trained to use DBL. This qualitative study sought to understand the experiences of one Peruvian statistics instructor and her students implementing DBL pedagogy and software. The DBL software was created to help instructors easily implement DBL. Findings identify specific benefits and challenges participants faced when using DBL. Benefits included the growth of students’ confidence in solving statistical problems and instructor satisfaction. Challenges included time-consuming instructor initial preparation and students’ learning transfer issues.

The FLAMINGO project: a comparison of galaxy cluster samples selected on mass, X-ray luminosity, Compton-Y parameter, or galaxy richness

ABSTRACT Galaxy clusters provide an avenue to expand our knowledge of cosmology and galaxy evolution. Because it is difficult to accurately measure the total mass of a large number of individual clusters, cluster samples are typically selected using an observable proxy for mass. Selection effects are therefore a key problem in understanding galaxy cluster statistics. We make use of the $(2.8~\rm {Gpc})^3$ FLAMINGO hydrodynamical simulation to investigate how selection based on X-ray luminosity, thermal Sunyaev–Zeldovich effect or galaxy richness influences the halo mass distribution. We define our selection cuts based on the median value of the observable at a fixed mass and compare the resulting samples to a mass-selected sample. We find that all samples are skewed towards lower mass haloes. For X-ray luminosity and richness cuts below a critical value, scatter dominates over the trend with mass and the median mass becomes biased increasingly low with respect to a mass-selected sample. At $z\le 0.5$, observable cuts corresponding to median halo masses between $M_\text{500c}=10^{14}$ and $10^{15}~\rm {{\rm M}_{\odot }}$ give nearly unbiased median masses for all selection methods, but X-ray selection results in biased medians for higher masses. For cuts corresponding to median masses $\lt 10^{14}$ at $z\le 0.5$ and for all masses at $z\ge 1$, only Compton-Y selection yields nearly unbiased median masses. Importantly, even when the median mass is unbiased, the scatter is not because for each selection the sample is skewed towards lower masses than a mass-selected sample. Each selection leads to a different bias in secondary quantities like cool-core fraction, temperature, and gas fraction.

Convergence of Peaceman-Rachford splitting method with Bregman distance for three-block nonconvex nonseparable optimization

Abstract It is of strong theoretical significance and application prospects to explore three-block nonconvex optimization with nonseparable structure, which are often modeled for many problems in machine learning, statistics, and image and signal processing. In this article, by combining the Bregman distance and Peaceman-Rachford splitting method, we propose a novel three-block Bregman Peaceman-Rachford splitting method (3-BPRSM). Under a general assumption, global convergence is presented via optimality conditions. Furthermore, we prove strong convergence when the augmented Lagrange function satisfies Kurdyka-Łojasiewicz property. In addition, if the association function possessing the Kurdyka-Łojasiewicz property exhibits a distinctive structure, then linear and sublinear convergence rate of 3-BPRSM can be guaranteed. Finally, a preliminary numerical experiment demonstrates the effectiveness.

Recognizing Statistical Problems in Reports of Clinical Trials: A Readers' Aid. Part 33 of a Series on Evaluation of Scientific Publications.

Readers of clinical trial reports should be able to critically evaluate the design, results, and conclusions of the trial. There are internationally accepted guidelines that define methodological standards for trial planning, statistical methods, and the display and interpretation of the results. Publications may nonetheless contain erroneous findings and interpretations. Statistical errors can arise in the planning of the trial, the analysis and display of the results, as well as in the interpretation of p-values and treatment effects in experimental and observational clinical trials. A useful aid for readers of medical publications should include a description of the potential statistical problems without complex theoretical background information. With this aim, we discuss certain major types of statistical error that the reader should be familiar with in order to be able to interpret the conclusions of these publications more easily. Statistical errors can already arise at an early stage through the choice of the wrong question to be addressed or the wrong population to be analyzed; such errors will inevitably have consequences. Before the start of any clinical trial, a primary endpoint must be defined, the sample size must be calculated, and the trial must be appropriately registered (among other requirements). With regard to the analysis, readers should for example take into account whether a statistical analysis plan with an intention-to-treat analysis existed for the study in question. They must be able to recognize erroneous methods of displaying and comparing data, confounding, as well as incorrect interpretations of p-values, and should take these problems into account when interpreting the findings. The problem of invalid causal inferences is not restricted to observational studies. Statistical errors do, indeed, arise. They should be detected as early as possible in various test instances. Nonetheless, readers should be able to judge independently whether the published clinical trial reflects meticulous and correct trial planning, appropriate display of the trial's results, and a proper, reasoned interpretation of the findings. The published checklists are a good aid for this purpose.

Bootstrapped Edge Count Tests for Nonparametric Two-Sample Inference Under Heterogeneity

Nonparametric two-sample testing is a classical problem in inferential statistics. While modern two-sample tests, such as the edge count test and its variants, can handle multivariate and non-Euclidean data, contemporary gargantuan datasets often exhibit heterogeneity due to the presence of latent subpopulations. Direct application of these tests, without regulating for such heterogeneity, may lead to incorrect statistical decisions. We develop a new nonparametric testing procedure that accurately detects differences between the two samples in the presence of unknown heterogeneity in the data generation process. Our framework handles this latent heterogeneity through a composite null that entertains the possibility that the two samples arise from a mixture distribution with identical component distributions but with possibly different mixing weights. In this regime, we study the asymptotic behavior of weighted edge count test statistic and show that it can be effectively recalibrated to detect arbitrary deviations from the composite null. For practical implementation we propose a Bootstrapped Weighted Edge Count test which involves a bootstrap-based calibration procedure that can be easily implemented across a wide range of heterogeneous regimes. A comprehensive simulation study and an application to detecting aberrant user behaviors in online games demonstrates the excellent non-asymptotic performance of the proposed test. Supplementary materials for this article are available online.

Statistically optimal firstorder algorithms: a proof via orthogonalization

Abstract We consider a class of statistical estimation problems in which we are given a random data matrix ${\boldsymbol{X}}\in{\mathbb{R}}^{n\times d}$ (and possibly some labels ${\boldsymbol{y}}\in{\mathbb{R}}^{n}$) and would like to estimate a coefficient vector ${\boldsymbol{\theta }}\in{\mathbb{R}}^{d}$ (or possibly a constant number of such vectors). Special cases include low-rank matrix estimation and regularized estimation in generalized linear models (e.g. sparse regression). Firstorder methods proceed by iteratively multiplying current estimates by ${\boldsymbol{X}}$ or its transpose. Examples include gradient descent or its accelerated variants. Under the assumption that the data matrix ${\boldsymbol{X}}$ is standard Gaussian, Celentano, Montanari, Wu (2020, Conference on Learning Theory, pp. 1078–1141. PMLR) proved that for any constant number of iterations (matrix vector multiplications), the optimal firstorder algorithm is a specific approximate message passing algorithm (known as ‘Bayes AMP’). The error of this estimator can be characterized in the high-dimensional asymptotics $n,d\to \infty $, $n/d\to \delta $, and provides a lower bound to the estimation error of any firstorder algorithm. Here we present a simpler proof of the same result, and generalize it to broader classes of data distributions and of firstorder algorithms, including algorithms with non-separable nonlinearities. Most importantly, the new proof is simpler, does not require to construct an equivalent tree-structured estimation problem, and is therefore susceptible of a broader range of applications.

Projected state ensemble of a generic model of many-body quantum chaos

Abstract The projected ensemble is based on the study of the quantum state of a subsystem A conditioned on projective measurements in its complement. Recent studies have observed that a more refined measure of the thermalization of a chaotic quantum system can be defined on the basis of convergence of the projected ensemble to a quantum state design, i.e. a system thermalizes when it becomes indistinguishable, up to the kth moment, from a Haar ensemble of uniformly distributed pure states. Here we consider a random unitary circuit with the brick-wall geometry and analyze its convergence to the Haar ensemble through the frame potential and its mapping to a statistical mechanical problem. This approach allows us to highlight a geometric interpretation of the frame potential based on the existence of a fluctuating membrane, similar to those appearing in the study of entanglement entropies. At large local Hilbert space dimension q, we find that all moments converge simultaneously with a time scaling linearly in the size of region A, a feature previously observed in dual unitary models. However, based on the geometric interpretation, we argue that the scaling at finite q on the basis of rare membrane fluctuations, finding the logarithmic scaling of design times t k = O ( log ⁡ k ) . Our results are supported with numerical simulations performed at q = 2.

Theory and Design of Audio Rooms: Physical Formulation

The time-dependent statistical chains of transition matrix B are successfully used to calculate the reverberation time TR and sound energy density field in audio rooms. This is a real breakthrough in the search for a statistical derivation of Sabines' imperial theory which has never been rigorously proven since 1898. Additionally, we provide proper design and calculations of sound wave energy density in audio rooms via two cuboid room examples. We provide proper design and calculations of sound wave energy density in audio rooms via two cuboid room examples. We also propose some general statistical rules that can be applied to statistical problems of physical phenomena in general, such as the thermal diffusion equation and the theory of sound waves in audio rooms. We also give a description of the theory and design of what we call comfortable sound rooms based on the purpose of their use via well-defined measurables. Finally, we provide a consistent derivation of Sabines' theory without using Ls=4V/A or any other statistical assumptions.

ANALISIS ATURAN ASOSIASI KESALAHAN STATISTIKA SISWA KELAS X-11 SMA NEGERI 10 SAMARINDA BERDASARKAN KRITERIA WATSON

Mathematics plays an important role in forming systematic thinking patterns, logic, and financial management skills. However, many students have difficulty in understanding mathematical concepts due to various factors, which affect the success of learning. This study aims to identify and classify common errors of class X-11 students of SMA Negeri 10 Samarinda in solving statistics problems using mixed quantitative and qualitative methods. solving statistics problems using mixed methods, namely quantitative and qualitative. Using the apriori algorithm with a minimum support of 20% and 80% confidence, four types of errors were found that were often made by students, namely incorrect data errors, procedural errors, missing conclusions, and skill hierarchy problems. Data analysis and interviews showed a strong correlation between these errors, where errors in data collection often lead to data manipulation and incorrect conclusions. To overcome this problem, the school has implemented more effective learning strategies, such as additional guidance, material adjustments, and contextual approaches. These strategies are expected to improve students' conceptual understanding, reduce errors, and improve academic achievement in statistics.