Statistical Problems Research Articles

Use of random integration to test equality of high dimensional covariance matrices.

Testing the equality of two covariance matrices is a fundamental problem in statistics, and especially challenging when the data are high-dimensional. Through a novel use of random integration, we can test the equality of high-dimensional covariance matrices without assuming parametric distributions for the two underlying populations, even if the dimension is much larger than the sample size. The asymptotic properties of our test for arbitrary number of covariates and sample size are studied in depth under a general multivariate model. The finite-sample performance of our test is evaluated through numerical studies. The empirical results demonstrate that our test is highly competitive with existing tests in a wide range of settings. In particular, our proposed test is distinctly powerful under different settings when there exist a few large or many small diagonal disturbances between the two covariance matrices.

Neutralise: An open science initiative for neutral comparison of two-sample tests.

The two-sample problem is one of the earliest problems in statistics: given two samples, the question is whether or not the observations were sampled from the same distribution. Many statistical tests have been developed for this problem, and many tests have been evaluated in simulation studies, but hardly any study has tried to set up a neutral comparison study. In this paper, we introduce an open science initiative that potentially allows for neutral comparisons of two-sample tests. It is designed as an open-source R package, a repository, and an online R Shiny app. This paper describes the principles, the design of the system and illustrates the use of thesystem.

Improved estimates for the number of non-negative integer matrices with given row and column sums

The number of non-negative integer matrices with given row and column sums features in a variety of problems in mathematics and statistics but no closed-form expression for it is known, so we rely on approximations. In this paper, we describe a new such approximation, motivated by consideration of the statistics of matrices with non-integer numbers of columns. This estimate can be evaluated in time linear in the size of the matrix and returns results of accuracy as good as or better than existing linear-time approximations across a wide range of settings. We show that the estimate is asymptotically exact in the regime of sparse tables, while empirically performing at least as well as other linear-time estimates in the regime of dense tables. We also use the new estimate as the starting point for an improved numerical method for either counting or sampling matrices with given margins using sequential importance sampling. Code implementing our methods is available.

Уголовно-правовая характеристика и проблемы исполнения уголовного наказания в виде обязательных работ

The article, according to the Judicial Department of the Supreme Court of the Russian Federation, provides an analysis of judicial practice in the appointment and execution of criminal punishment in the form of compulsory labor. The problematic issues of the execution of this type of punishment are currently being considered. Special attention is paid to the consideration of problems related to the malicious evasion of convicts from serving their sentences in the form of compulsory labor. The essence of the punishment under study is analyzed. The subject of the study was statistical data on compulsory work and legal literature on the issues of considering the problems of the practice of assigning and executing this criminal punishment. Purpose. The purpose of the study is to highlight the Russian practice of imposing and executing punishment in the form of compulsory labor, identifying problems and finding ways to solve them. The conclusions formulated in the article will make it possible to increase the effectiveness of the activities of criminal enforcement inspections for the execution of this criminal punishment, and will also serve as a source material for subsequent research in the field of the execution of criminal penalties without isolating convicts from society. Methodology. The goal was achieved using dialectical methods of scientific cognition of both general and private scientific research methods: formal legal, comparative legal, descriptive, functional and scientific analysis method. Results. The analysis of statistical data, theoretical and practical problems of criminal law norms regulating the appointment and execution of compulsory work, allowed us to focus on a wide range of problems both in the criminal law sphere and in the relationship of other normative legal acts with criminal and penal enforcement legislation on the regulation of compulsory work. The materials of the article will be of interest to both scientists and practitioners dealing with the problems of execution of alternative types of punishment.

Application of fused graphical lasso to statistical inference for multiple sparse precision matrices.

In this paper, the fused graphical lasso (FGL) method is used to estimate multiple precision matrices from multiple populations simultaneously. The lasso penalty in the FGL model is a restraint on sparsity of precision matrices, and a moderate penalty on the two precision matrices from distinct groups restrains the similar structure across multiple groups. In high-dimensional settings, an oracle inequality is provided for FGL estimators, which is necessary to establish the central limit law. We not only focus on point estimation of a precision matrix, but also work on hypothesis testing for a linear combination of the entries of multiple precision matrices. We apply a de-biasing technology, which is used to obtain a new consistent estimator with known distribution for implementing the statistical inference, and extend the statistical inference problem to multiple populations. The corresponding de-biasing FGL estimator and its asymptotic theory are provided. A simulation study and an application of the diffuse large B-cell lymphoma data show that the proposed test works well in high-dimensional situation.

Application of maximum statistical entropy in formulating a non-gaussian probability density function in flow uncertainty analysis with prior measurement knowledge

In mechanical, civil and chemical engineering systems the accuracies of flow measurement instruments is conventionally specified by certified measurement capabilities (CMCs) that are symmetric, however it is physically possible for some flow instruments and equipment to exhibit asymmetric non-Gaussian behaviour. In this paper the influence of non-Gaussian uncertainties is investigated using direct Monte Carlo simulations to construct a probability density function (PDF) using representative non-Gaussian surface roughness data for a commercial steel pipe friction factor. Actual PDF results are compared and contrasted with a symmetric Gaussian PDF, and reveal inconsistencies in the statistical distributions that cannot be neglected in high accuracy flow measurements. The non-Gaussian PDF is visualized with a kernel density estimate (KDE) scheme to infer an initial qualitative shape of the actual PDF using the approximate locations of the normalized peaks as a initial metrologist estimate of the measurement density. This is then utilized as inputs in a maximum statistical entropy functional to optimize the actual non-Gaussian PDF using a nonlinear optimization of Lagrange multipliers for a mathematically unique PDE. Novelties in the present study is that a new methodology has been developed for statistical sampling from non-monotonic non-Gaussian distributions with accompanying Python and Matlab/GNU Octave computer codes, and a new methodology for utilizing metrologist's expert prior knowledge of PDF peaks and locations for constructing an a priori estimate of the shape of unknown density have been incorporated into the maximum statistical entropy nonlinear optimization problem for a faster and more efficient approach for generating statistical information and insights in constructing high accuracy non-Gaussian PDFs of real world messy engineering measurements.

Robust Optimization and Inference on Manifolds

We propose a robust and scalable procedure for general optimization and inference problems on manifolds leveraging the classical idea of `median-of-means' estimation. This is motivated by ubiquitous examples and applications in modern data science in which a statistical learning problem can be cast as an optimization problem over manifolds. Being able to incorporate the underlying geometry for inference while addressing the need for robustness and scalability presents great challenges. We address these challenges by first proving a key lemma that characterizes some crucial properties of geometric medians on manifolds. In turn, this allows us to prove robustness and tighter concentration of our proposed final estimator in a subsequent theorem. This estimator aggregates a collection of subset estimators by taking their geometric median over the manifold. We illustrate bounds on this estimator via calculations in explicit examples. The robustness and scalability of the procedure is illustrated in numerical examples on both simulated and real data sets.

A Data-Driven Framework for Power System Event Type Identification via Safe Semi-Supervised Techniques

This paper investigates the use of phasor measurement unit (PMU) data with deep learning techniques to construct real-time event identification models for transmission networks. Increasing penetration of distributed energy resources represents a great opportunity to achieve decarbonization, as well as challenges in systematic situational awareness. When high-resolution PMU data and sufficient manually recorded event labels are available, the power event identification problem is defined as a statistical classification problem that can be solved by numerous cutting-edge classifiers. However, in real grids, collecting tremendous high-quality event labels is quite expensive. Utilities frequently have a large number of event records without in-depth details (i.e., unlabeled events). To bridge this gap, we propose a novel semi-supervised learning-based method to improve the performance of event classifiers trained with a limited number of labeled events by exploiting the information from massive unlabeled events. In other words, compared to existing data-driven methods, our method requires only a small portion of labeled data to achieve a similar level of accuracy. Meanwhile, this work discusses and addresses the performance degradation caused by class distribution mismatch between the training set and the real applications. Specifically, this method utilizes pseudo-labeling technique to investigate the value of unlabeled events and incrementally expands the training dataset. Moreover, a safe learning mechanism is developed to mitigate the impacts of class distribution mismatch and prevent performance degradation. Based on the proposed safe learning mechanism, our model does not directly use all unlabeled events during model training, but selectively uses them through a comprehensive evaluation procedure. Numerical studies on a sizable PMU dataset have been used to validate the performance of the proposed method.

基于Matlab的概率论与数理统计问题的探究与算法

基于Matlab的概率论与数理统计问题的探究与算法

Технология оптимальной обработки сигналов в области ультразвука с высоким разрешением

Purpose of the work: To present the technology of processing ultrasonic signals with high resolution. Methods: The basis is the Helstrom substitution, which eliminates the ambiguity of maximizing the likelihood function in terms of reception time, it is shown that the transformed likelihood function and the transformed likelihood ratio functional as a result of Helstrom substitution, mutually complementing each other, allow solving statistical problems in the field of high-resolution signals. Results: Based on model calculations, the result of the Helstrom substitution is presented, eliminating the ambiguity of maximizing the likelihood function. Statistics of amplitudes and reception times of two ultrasonic signals are given depending on the signal-to-noise ratio and depending on the difference in reception times. Estimates of the operating resolution range of two ultrasonic signals and estimates of the signal-to-noise ratio range in which the errors in the estimates of signal parameters do not exceed 10 % are given.

An Efficient Greedy Search Algorithm for High-dimensional Linear Discriminant Analysis.

High-dimensional classification is an important statistical problem that has applications in many areas. One widely used classifier is the Linear Discriminant Analysis (LDA). In recent years, many regularized LDA classifiers have been proposed to solve the problem of high-dimensional classification. However, these methods rely on inverting a large matrix or solving large-scale optimization problems to render classification rules-methods that are computationally prohibitive when the dimension is ultra-high. With the emergence of big data, it is increasingly important to develop more efficient algorithms to solve the high-dimensional LDA problem. In this paper, we propose an efficient greedy search algorithm that depends solely on closed-form formulae to learn a high-dimensional LDA rule. We establish theoretical guarantee of its statistical properties in terms of variable selection and error rate consistency; in addition, we provide an explicit interpretation of the extra information brought by an additional feature in a LDA problem under some mild distributional assumptions. We demonstrate that this new algorithm drastically improves computational speed compared with other high-dimensional LDA methods, while maintaining comparable or even better classification performance.

An instance of negative variance-like ‘components’ in optics by way of applying multivariable calculus to a statistical problem

In the field of optics, the design of a lens often involves analyses of the variability in an optical parameter (such as lens focal length) under manufacturing conditions, which is done in order to identify sources of high variability that may cause the optical parameter to be out of its specification. In this interdisciplinary paper, the author shares his experience in lens design, where analyses of variability are often performed through examining the partial derivatives in a multivariable Taylor series expansion of an optical parameter to second order when interactions are expected. It is shown algebraically that when the second-order expansion is applied to express a sum of squares, negative terms in the sum can occur. This instance of negative-valued “variance-like components” is interesting pedagogically in applied mathematics and statistics, as well as from an engineering standpoint, as it suggests that, based on theory, the variance of a function of random variables may not necessarily be comprised of a simple sum of positive-valued squared deviations comprising its total variance.

Моделирование «ковидного провала» плановой госпитализации в период разгара пандемии COVID-19: анализ и оценка потерь использованной коечной мощности специализированного оториноларингологического стационара

Introduction. In the midst of the COVID-19 pandemic, healthcare organizations around the world were forced to change their operating hours, and some, especially those providing specialized medical care, were forced to stop operating completely. The purpose of the study is to develop a year-by-year model of the movement of patients before the pandemic and at its height, based on long-term data on the movement of inpatients. With its help, analyze possible losses in bed capacity (expressed in the number of idle bed days, or unserved patients) from the pandemic during 2020 — the year of the height of the pandemic. Materials and methods. The study was carried out according to the design of an observational retrospective analytical study. As research materials, we used anonymized data on the number of patients in the hospital at the Federal State Budgetary Institution «St. Petersburg Research Institute of ENT» of the Ministry of Health of Russia, obtained from the medical registration form N 007/u-02 „Sheet of daily records of the movement of patients and the bed capacity of a round-the-clock hospital, day hospital at a hospital facility“). The R and RStudio software environment was used as the main software product for solving statistical problems. Results. A load model has been created (the number of daily inpatients undergoing treatment) during the year. An analytical model of the “Covid failure” of bed capacity utilization during the height of the pandemic has been created. The coordinates of the dynamic points of bed release and return to typical bed occupancy values for this period of time were calculated (using linear regression forecasting). The share of hospital losses during the pandemic was calculated and assessed. Discussion. The use of “Covid failure” data superimposed on “pre-Covid” indicators allows not only to estimate annual losses, but also to predictively minimize losses during planned closures of hospital beds, for example, for repairs, given that the resumption of work of a medical institution’s beds will often be slower than the speed at which work stops. Analysis of linear regression coefficients of bed release and occupancy rates can give an idea of the speed of reaching standard values for the volume of medical care. As studies have shown during the COVID-19 pandemic, the minimum value of the quarantine period after the complete vacancy of beds is 14 days, and for high-risk scenarios 21 days. Based on this, for the conditions of organizing medical otorhinolaryngological care, the minimum predicted value of the time from the closure of hospitalization to its opening may be about 37 days after the start of patient discharge. The approach and mathematical functionality presented in the article can be used on the initial data of other institutions, as well as in preparation for the “Disease X” pandemic.

Opportunities and Challenges of Remote Sensing, Geospatial Data, and Machine Learning in Obtaining Accessibility and Location Information for Sustainable Development in Indonesia

With the advancement of technologies so does the data collection method which creates a large, rapid, and diverse stream of data. Statistic Indonesia (BPS) has also encouraged to utilize this by starting to collect geospatial information on respondents and public facilities. To keep up with this a change needs to be made in processing methods to accommodate massive, high-dimensional, and multiform data collected in different forms such as machine learning. This progression also opens up a new opportunity for tackling various statistical data problems such as accessibility and location data. Remote sensing is one of the big data sources that undergoes a lot of changes shown in the high spatial and temporal resolution satellite imagery availability, together with the BPS geotagging data shows great promise in classifying land use and geospatial analysis. Even so, there are still some challenges in remote sensing as well as other geospatial data utilization. The goals of this review paper are to study the opportunities and challenges in utilizing remote sensing, geospatial data, and machine learning for accessibility and location information. In this paper, we explore the possibilities and limitations in its implementation into SDGs indicators that involve accessibility and location such as indicators 9.1.1, 11.1.1, 11.2.1, 11.3.1, and 11.7.1 including other variables needed for the calculation like access to public facilities. Moreover, our experiment using geotagging data shows potential in improving proportion estimation when compared to using a simple ratio. Our DEGURBA following the UN definition using machine learning LULC for dasymetric mapping also provides more insight compared to the existing data. We can conclude that there are great opportunities in applying remote sensing and other geospatial data to monitor the accessibility and location to further sustainable development in Indonesia.

Neural Coding as a Statistical Testing Problem

We take the testing perspective to understand what the minimal discrimination time between two stimuli is for different types of rate coding neurons. Our main goal is to describe the testing abilities of two different encoding systems: place cells and grid cells. In particular, we show, through the notion of adaptation, that a fixed place cell system can have a minimum discrimination time that decreases when the stimuli are further away. This could be a considerable advantage for the place cell system that could complement the grid cell system, which is able to discriminate stimuli that are much closer than place cells.

MODERN NUMERICAL METHODS OF ANALYSIS OF STATISTICAL DATA IN THE SCHOOL COURSE

Mathematical modelling of practical statistical problems is a current topic in the school course, related to the development of various approaches and methods of modern solutions to many practical problems. The purpose of mathematical statistics is the development of methods of statistical observation and the collection and analysis of statistical data. The given article shows several numerical methods for studying mathematical statistics in the school course: measurements of central tendency, measurements of position, measurements of dispersion. Measures of central tendency indicate the “center” of the data along a number line and are usually reported as values representing the data. There are three common measures of central tendency: the arithmetic mean, the median, and the mode. Position metrics represent the three most basic positions in a list of numeric data, ordered from smallest to largest: start, end, and middle. In addition to these, the most common measures of position, quartiles and percentages, are also presented. Measures of dispersion show the degree to which the data are spread out. Among the measures of dispersion of the statistic, the rank, interquartile range, and standard deviation are presented. The practice of solving statistical problems serves to develop the skills of using probabilistic statistical concepts and methods at school, so the work presents the brief content of each method, calculation formulas, and examples.

Quantile Regression Model and Its Application Research

Quantile regression is a regression analysis method that estimates the parameters of a model by minimizing the weighted sum of absolute residuals. It was introduced by Roger Koenker in 1978. As a complementary and extended approach to the traditional regression method, namely least squares method, quantile regression addresses the limitations of least squares method in the presence of heteroscedasticity and ensures the robustness of quantile regression through its robustness to outliers, which compensates for the weakness of least squares method in dealing with outlier data. In practical applications, quantile regression can provide a more comprehensive reflection of data information and capture the tails of the distribution of the dependent variable, thus overcoming the limitation of least squares method that can only estimate the central tendency of the dependent variable distribution. Moreover, quantile regression provides more reasonable interpretations and prevents biased or even erroneous estimations that can occur when using least squares method. Overall, quantile regression offers a novel regression method that addresses many shortcomings of least squares method. By combining quantile regression with least squares method, we can understand both the central tendency and tail behavior of the dependent variable distribution. The fitting results obtained from quantile regression can also provide insights into the suitability of least squares estimation. Briefly, the combination of both methods yields better results in statistical problems. This paper introduces the principles of quantile regression and further discusses its scope and application, aiming to provide a preliminary summary for a better understanding of quantile regression.

Adaptive minimax optimality in statistical inverse problems via SOLIT—Sharp Optimal Lepskiĭ-Inspired Tuning

We consider statistical linear inverse problems in separable Hilbert spaces and filter-based reconstruction methods of the form fˆα=qαT∗TT∗Y , where Y is the available data, T the forward operator, qαα∈A an ordered filter, and α > 0 a regularization parameter. Whenever such a method is used in practice, α has to be appropriately chosen. Typically, the aim is to find or at least approximate the best possible α in the sense that mean squared error (MSE) E[∥fˆα−f†∥2] w.r.t. the true solution f† is minimized. In this paper, we introduce the Sharp Optimal Lepskiĭ-Inspired Tuning (SOLIT) method, which yields an a posteriori parameter choice rule ensuring adaptive minimax rates of convergence. It depends only on Y and the noise level σ as well as the operator T and the filter qαα∈A and does not require any problem-dependent tuning of further parameters. We prove an oracle inequality for the corresponding MSE in a general setting and derive the rates of convergence in different scenarios. By a careful analysis we show that no other a posteriori parameter choice rule can yield a better performance in terms of the order of the convergence rate of the MSE. In particular, our results reveal that the typical understanding of Lepskiĭ-type methods in inverse problems leading to a loss of a log factor is wrong. In addition, the empirical performance of SOLIT is examined in simulations.

Confidence in causal inference under structure uncertainty in linear causal models with equal variances

Abstract Inferring the effect of interventions within complex systems is a fundamental problem of statistics. A widely studied approach uses structural causal models that postulate noisy functional relations among a set of interacting variables. The underlying causal structure is then naturally represented by a directed graph whose edges indicate direct causal dependencies. In a recent line of work, additional assumptions on the causal models have been shown to render this causal graph identifiable from observational data alone. One example is the assumption of linear causal relations with equal error variances that we will take up in this work. When the graph structure is known, classical methods may be used for calculating estimates and confidence intervals for causal-effects. However, in many applications, expert knowledge that provides an a priori valid causal structure is not available. Lacking alternatives, a commonly used two-step approach first learns a graph and then treats the graph as known in inference. This, however, yields confidence intervals that are overly optimistic and fail to account for the data-driven model choice. We argue that to draw reliable conclusions, it is necessary to incorporate the remaining uncertainty about the underlying causal structure in confidence statements about causal-effects. To address this issue, we present a framework based on test inversion that allows us to give confidence regions for total causal-effects that capture both sources of uncertainty: causal structure and numerical size of non-zero effects.

Neuro-symbolic recommendation model based on logic query

Recommendation systems assist users in finding items that are relevant to them. However, recommendation is not only an inductive statistics problem using data; it is also a cognitive task of reasoning decisions based on knowledge extracted from information. Hence, a logic system can naturally be incorporated for reasoning in a recommendation task. Although logic-based hard-rule approaches can provide a powerful reasoning ability, they struggle to cope with inconsistent and incomplete knowledge in real-world tasks, especially for complex tasks such as recommendation. To address these issues, we propose a neuro-symbolic recommendation model, which transforms user history interactions into a logic expression and then transforms the recommendation prediction into a query task based on this logic expression. The logic expressions are then computed based on the modular logic operations of the neural network. We also construct an implicit logic encoder to reasonably reduce the complexity of the logic computation. Finally, a user’s interest items can be queried in the vector space based on the computation results. Experiments on three well-known datasets verified that the proposed method outperforms state of the art shallow, deep, session, and reasoning models.