Tail Of Distribution Research Articles

Statistical inference from partially nominated sets: An application to estimating the prevalence of osteoporosis among adult women

This paper focuses on drawing statistical inference based on a novel variant of maxima or minima nomination sampling (NS) designs. These sampling designs are useful for obtaining more representative sample units from the tails of the population distribution using the available auxiliary ranking information. However, one common difficulty in performing NS in practice is that the researcher cannot obtain a nominated sample unless he/she uniquely determines the sample unit with the highest or the lowest rank in each set. To overcome this problem, a variant of NS, which is called partial nomination sampling, is proposed, in which the researcher is allowed to declare that two or more units are tied in the ranks whenever he/she cannot find the sample unit with the highest or the lowest rank. Based on this sampling design, two asymptotically unbiased estimators are developed for the cumulative distribution function, which is obtained using maximum likelihood and moment-based approaches, and their asymptotic normalities are proved. Several numerical studies have shown that the proposed estimators have higher relative efficiencies than their counterparts in simple random sampling in analyzing either the upper or the lower tail of the parent distribution. The procedures that we developed are then implemented on a real dataset from the Third National Health and Nutrition Examination Survey (NHANES III) to estimate the prevalence of osteoporosis among adult women aged 50 and over. It is shown that in certain circumstances, the techniques that we have developed require only one-third of the sample size needed in SRS to achieve the desired precision. This results in a considerable reduction in time and cost compared to the standard SRS method.

On the asymptotic properties of extreme values of discrete random variables

The article examines the almost sure asymptotic behaviour of extreme values of discrete random variables. We consider the case of random variables whose tails of the distribution are close to the tails of the geometric distribution or decrease faster.

AI models collapse when trained on recursively generated data

Stable diffusion revolutionized image creation from descriptive text. GPT-2 (ref. 1), GPT-3(.5) (ref. 2) and GPT-4 (ref. 3) demonstrated high performance across a variety of language tasks. ChatGPT introduced such language models to the public. It is now clear that generative artificial intelligence (AI) such as large language models (LLMs) is here to stay and will substantially change the ecosystem of online text and images. Here we consider what may happen to GPT-{n} once LLMs contribute much of the text found online. We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear. We refer to this effect as ‘model collapse’ and show that it can occur in LLMs as well as in variational autoencoders (VAEs) and Gaussian mixture models (GMMs). We build theoretical intuition behind the phenomenon and portray its ubiquity among all learned generative models. We demonstrate that it must be taken seriously if we are to sustain the benefits of training from large-scale data scraped from the web. Indeed, the value of data collected about genuine human interactions with systems will be increasingly valuable in the presence of LLM-generated content in data crawled from the Internet.

Multi‐plant firms and the heavy tail of firm size distribution

AbstractThe right tail of the firm size distribution has a heavy tail. The origin of this phenomenon, especially the specific characteristics of firms driving this pattern, remain a subject of extensive debate. Previous work has shown that plant size distribution has thinner tails than firm size distribution, indicating the role of multi‐plant firms. However, we do not know whether this phenomenon is simply a mechanical effect arising from aggregation across multiple plants or whether the plants of multi‐plant firms are different from those of single‐plant firms. Using novel data with plant‐to‐firm mapping, we document that plants of multi‐plant firms are more heavy‐tailed than single‐plant firms, indicating the dominance of the selection effect at the intensive margin. Extensive margin via aggregation of sales at the firm level plays a less crucial role than the selection effect. Importantly, single‐plant exporters have a thinner tail than multi‐plant non‐exporters, suggesting a more dominant role of multi‐plant identity than export identity in explaining heavy tails.

Comparison of outlier detection approaches for compressive strength of cement-based mortars

EN 196-1 states that the compressive strength result of testing of standard mortar specimens corresponds to the average of the six individual measurements. If one of the six individual measurements varies more than ±10 % from the average, it must be excluded, and a new average is calculated with the five remaining values. This methodology might be interesting for quality control or characterization of cement; however, it may not be the best approach for research or scientific purposes. Applying EN 196-1 to non-standard cement-based mortars with multiple and even unconventional supplementary cementitious materials may induce even more variation in the result. Also, mixtures with higher mechanical strength generally present the highest absolute variation. The main goal of the present work is to compare different methodologies, such as Tails of Normal Distribution, Thompson-Tau, Box-plot, T-test, Dixon Criteria, and other methods provided by engineering standards, to evaluate the existence of discrepant values and the effect of their exclusion (or not) on compressive strength experimental values, namely for science purposes. A sample of compressive strength of 338 mortar specimens was used The main findings revealed that excluding data points as a result of applying 8 of the 13 methods under study did not affect the mechanical strength result by more than ±5 %. The method described in the Brazilian standard that guides preparation, control, receipt, and acceptance for Portland cement concrete, would invalidate almost 46 % of all mixtures produced and did not prove to be reliable. Simplified methods of excluding outliers in relative or absolute percentages of 6 % around the average value can be considered an interesting option for professionals involved in laboratory testing of building materials. Otherwise, the authors would recommend applying the either Dixon Criteria or T test of the ASTM E178 for scientific purposes.

How Do Quantitative Videofluoroscopy Measures Differ Between People With Amyotrophic Lateral Sclerosis and Age-Matched Healthy Adults?

Dysphagia is a leading cause of morbidity in people with amyotrophic lateral sclerosis (PwALS). Previous videofluoroscopic swallowing studies (VFSS) in PwALS do not account for the influence of senescence. We aimed to compare swallowing in PwALS and an age- and sex-matched control group using healthy reference data to define typical and atypical values. We conducted retrospective analysis of VFSS data from 19 PwALS (10 male, Mage = 63 years, range: 47-82) compared to control data from a cohort of healthy adults. Participants swallowed 20% w/v liquid barium from thin to extremely thick consistency. Blinded duplicate VFSS analysis using the ASPEKT (Analysis of Swallowing Physiology: Events, Kinematics and Timing) method yielded descriptive statistics for 16 quantitative VFSS parameters by consistency. Mann-Whitney U tests were used to identify significant cohort differences. Additionally, the frequencies of atypical values (in the 25% tails of the reference distribution) were tabulated by cohort and compared using odds ratios. PwALS showed increased frequencies of multiple swallows per bolus, incomplete laryngeal vestibule closure, and reduced hyoid speed across consistencies. By contrast, similar frequencies of atypical values for pharyngeal constriction and residue in both cohorts suggest that age-related changes may contribute to the presence of these features in PwALS. This analysis builds on previous descriptions of swallowing pathophysiology in amyotrophic lateral sclerosis (ALS) by clarifying the extent to which aging may account for some of the atypical findings seen in this patient population. Longitudinal studies are recommended to further differentiate the effects of ALS from age-related changes in swallowing over the course of disease progression.

Extreme weather, policy uncertainty, and risk spillovers between energy, financial, and carbon markets

Extreme weather, as well as policy uncertainties associated with climate change, have fueled global energy market turbulence and cross-market risk contagion, which have had serious effects on the economy. Using time-frequency network analysis based on quantile vector autoregression (QVAR), this study reveals the risk spillovers between China's energy, financial, and carbon markets in different time and frequency domains. The effects of extreme weather and policy uncertainty on market risk spillovers are also examined. The results show that: (1) Risk spillovers vary significantly in different time and frequency domains, and extreme risks tend to occur in the tails of the shock distribution, and market risk spillovers occur mainly in the long term; (2) Extreme weather and policy uncertainty significantly affect energy, financial, and carbon markets, altering their net spillover levels. Among which the coal market is the most volatile and highly sensitive to changes in external conditions; (3) Risk spillovers exhibit distinct time-varying characteristics, with extreme weather, climate policy uncertainty, and trade policy uncertainty exacerbating market risk spillovers. These findings could serve as a reference for monitoring market risk spillovers, facilitating energy transition, and achieving dual‑carbon targets.

Beyond Zipf’s law: Exploring the discrete generalized beta distribution in open-source repositories

Rank-size distributions, such as Zipf’s Law, have been instrumental in providing insights into the emergence of hierarchies across diverse systems, from linguistic corpuses to urban structures. However, the application of Zipf’s Law reveals limitations, particularly in its focus on distribution tails, sometimes overlooking a large proportion of the data which might play a pivotal role in system dynamics. Yet, fitting rank-size distributions other than a straight line on the log–log scale requires caution. In this study, we re-evaluate the utility of rank-size distributions by contrasting the traditional Zipf’s Law with the Discrete Generalized Beta Distribution (DGBD). We show the need of cautious fitting techniques for rank distributions, including the use of binning to prevent overfitting to data tails. Through both analytical derivation and empirical validation on commit data of open-source repositories, we show that DGBD consistently improves over Zipf distribution for concave rank distributions of large datasets (N≥100). This approach contributes to the advancement of methodologies for analyzing hierarchical systems.

Merging TROPOMI and eddy covariance observations to quantify 5-years of daily CH4 emissions over coal-mine dominated region

A simple and flexible mass balance approach was applied to observations of XCH4 from TROPOMI to estimate CH4 emissions over Shanxi Province, including the impacts of advective transport, pressure transport, and atmospheric diffusion. High-frequency eddy-covariance flux observations were used to constrain the driving terms of the mass balance equation. This equation was then used to calculate day-to-day and 5 km × 5 km grided CH4 emissions from May 2018 to July 2022 based on TROPOMI RPRO column CH4 observations. The Shanxi-wide emissions of CH4, 126 ± 58.8 ug/m2/s, shows a fat tail distribution and high variability on a daily time scale (the 90th percentile is 2.14 times the mean and 2.74 times the median). As the number of days in the rolling average increases, the change in the variation decreases to 128 ± 35.7 ug/m2/s at 10-day, 128 ± 19.8 ug/m2/s at 30-day and 127 ± 13.9 ug/m2/s at 90-day. The range of values of the annual mean emissions on coal mine grids within Shanxi for the years 2018 to 2022 was 122 ± 58.2, 131 ± 71.2, 111 ± 63.6, 129 ± 87.1, and 138 ± 63.4 ug/m2/s, respectively. The 5-year average emissions from TROPOMI are 131 ± 68.0 ug/m2/s versus 125 ± 94.6 ug/m2/s on the grids where the EDGAR bottom-up database also has data, indicating that those pixels with mines dominate the overall emissions in terms of both magnitude and variability. The results show that high-frequency observation-based campaigns can produce a less biased result in terms of both the spatial and temporal distribution of CH4 emissions as compared with approaches using either low-frequency data or bottom-up databases, that coal mines dominate the sources of CH4 in Shanxi, and that the observed fat tail distribution can be accounted for using this approach.

Estimating changepoints in extremal dependence, applied to aviation stock prices during COVID-19 pandemic

The dependence in the tails of the joint distribution of two random variables is generally assessed using χ-measure, the limiting conditional probability of one variable being extremely high given the other variable is also extremely high. This work is motivated by the structural changes in χ-measure between the daily rate of return (RoR) of the two Indian airlines, IndiGo and SpiceJet, during the COVID-19 pandemic. We model the daily maximum and minimum RoR vectors (potentially transformed) using the bivariate Hüsler-Reiss (BHR) distribution. To estimate the changepoint in the χ-measure of the BHR distribution, we explore two changepoint detection procedures based on the Likelihood Ratio Test (LRT) and Modified Information Criterion (MIC). We obtain critical values and power curves of the LRT and MIC test statistics for low through high values of χ-measure. We also explore the consistency of the estimators of the changepoint based on LRT and MIC numerically. In our data application, for RoR maxima and minima, the most prominent changepoints detected by LRT and MIC are close to the announcement of the first phases of lockdown and unlock, respectively, which are realistic; thus, our study would be beneficial for portfolio optimization in the case of future pandemic situations.

Bounds for the distribution and moments of the forward and backward recurrence times in a renewal process

We obtain a variety of lower and upper bounds for the bivariate tail of the joint distribution between the (forward and backward) recurrence times in a renewal process. Some of these bounds are valid generally, some others in the case where the interarrival distribution F exhibits monotone aging. We also give various bounds for the moments of the recurrence times. Our results improve upon existing bounds for these quantities in the literature. We also discuss the case where the aging of F is non-monotonic. Finally, we present numerical examples to illustrate the performance of the new bounds.

Alternative statistical interpretation for the apparent plateaus in the duration distributions of gamma-ray bursts

Context. The existence of a plateau in the short-duration tail of the observed distribution of cosmological long-soft gamma ray bursts (LGRBs) is posited to be the first direct evidence of collapsars. A similar plateau in the short-duration tail of the observed duration distribution of short-hard gamma ray bursts (SGRBs) has been suggested as evidence of compact binary mergers. Aims. We present an equally plausible alternative interpretation for this evidence, based on a purely statistical approach. Methods. Specifically, we show that the observed plateau in the short-duration tail of the duration distribution of LGRBs can naturally occur in the statistical distributions of strictly positive physical quantities, exacerbated by the effects of mixing with the duration distribution of SGRBs, observational selection effects, and data aggregation (e.g., binning) methodologies. The observed plateau in the short-duration tail of the observed distributions of SGRBs may likewise result from a combination of sample incompleteness and inhomogeneous binning of data. We further confirm the impact of these factors on the observation of a plateau in the duration distributions of GRBs through extensive numerical Monte Carlo simulations. Results. This analysis corroborates and strengthens a purely statistical and sample-incompleteness interpretation of the observed plateau in the duration distribution of LGRBs and SGRBs, without invoking the physics of collapsars or jet-propagation through the stellar envelope.

Modeling Riverbed Elevation and Bedload Tracer Transport Resting Times Using Fractional Laplace Motion

AbstractRiverbed elevations play a crucial role in sediment transport and flow resistance, making it essential to understand and quantify their effects. This knowledge is vital for various fields, including river engineering and stream ecology. Previous observations have revealed that fluctuations in the bed surface can exhibit both multifractal and monofractal behaviors. Specifically, the probability distribution function (PDF) of elevation increments may transition from Laplace (two‐sided exponential) to Gaussian with increasing scales or consistently remain Gaussian, respectively. These differences at the finest timescale lead to distinct patterns of bedload particle exchange with the bed surface, thereby influencing particle resting times and streamwise transport. In this paper, we utilize the fractional Laplace motion (FLM) model to analyze riverbed elevation series, demonstrating its capability to capture both mono‐ and multi‐fractal behaviors. Our focus is on studying the resting time distribution of bedload particles during downstream transport, with the FLM model primarily parameterized based on the Laplace distribution of increments PDF at the finest timescale. Resting times are extracted from the bed elevation series by identifying pairs of adjacent deposition and entrainment events at the same elevation. We demonstrate that in cases of insufficient data series length, the FLM model robustly estimates the tail exponent of the resting time distribution. Notably, the tail of the exceedance probability distribution of resting times is much heavier for experimental measurements displaying Laplace increments PDF at the finest scale, compared to previous studies observing Gaussian PDF for bed elevation.

Storm Gust Prediction with the Integration of Machine Learning Algorithms and WRF Model Variables for the Northeast United States

Abstract Wind gusts are often associated with severe hazards and can cause structural and environmental damages, making gust prediction a crucial element of weather forecasting services. In this study, we explored the utilization of machine learning (ML) algorithms integrated with numerical weather prediction outputs from the Weather Research and Forecasting (WRF) Model, to align the estimation of wind gust potential with observed gusts. We have used two ML algorithms, namely, random forest (RF) and extreme gradient boosting (XGB), along with two statistical techniques: generalized linear model with identity link function (GLM-Identity) and generalized linear model with log link function (GLM-Log), to predict storm wind gusts for the northeast (NE) United States. We used 61 simulated extratropical and tropical storms that occurred between 2005 and 2020 to develop and validate the ML and statistical models. To assess the ML model performance, we compared our results with postprocessed gust potential from WRF. Our findings showed that ML models, especially XGB, performed significantly better than statistical models and Unified Post Processor for the WRF (WRF-UPP) Model and were able to better align predicted with observed gusts across all storms. The ML models faced challenges capturing the upper tail of the gust distribution, and the learning curves suggested that XGB was more effective than RF in generating better predictions with fewer storms.

WRF-Comfort: simulating microscale variability in outdoor heat stress at the city scale with a mesoscale model

Abstract. Urban overheating and its ongoing exacerbation due to global warming and urban development lead to increased exposure to urban heat and increased thermal discomfort and heat stress. To quantify thermal stress, specific indices have been proposed that depend on air temperature, mean radiant temperature (MRT), wind speed, and relative humidity. While temperature and humidity vary on scales of hundreds of meters, MRT and wind speed are strongly affected by individual buildings and trees and vary on the meter scale. Therefore, most numerical thermal comfort studies apply microscale models to limited spatial domains (commonly representing urban neighborhoods with building blocks) with resolutions on the order of 1 m and a few hours of simulation. This prevents the analysis of the impact of city-scale adaptation and/or mitigation strategies on thermal stress and comfort. To solve this problem, we develop a methodology to estimate thermal stress indicators and their subgrid variability in mesoscale models – here applied to the multilayer urban canopy parameterization BEP-BEM within the Weather Research and Forecasting (WRF) model. The new scheme (consisting of three main steps) can readily assess intra-neighborhood-scale heat stress distributions across whole cities and for timescales of minutes to years. The first key component of the approach is the estimation of MRT in several locations within streets for different street orientations. Second, mean wind speed and its subgrid variability are downscaled as a function of the local urban morphology based on relations derived from a set of microscale LES and RANS simulations across a wide range of realistic and idealized urban morphologies. Lastly, we compute the distributions of two thermal stress indices for each grid square, combining all the subgrid values of MRT, wind speed, air temperature, and absolute humidity. From these distributions, we quantify the high and low tails of the heat stress distribution in each grid square across the city, representing the thermal diversity experienced in street canyons. In this contribution, we present the core methodology as well as simulation results for Madrid (Spain), which illustrate strong differences between heat stress indices and common heat metrics like air or surface temperature both across the city and over the diurnal cycle.

Getting the right tail right: Modeling tails of health expenditure distributions

Health expenditure data almost always include extreme values, implying that the underlying distribution has heavy tails. This may result in infinite variances as well as higher-order moments and bias the commonly used least squares methods. To accommodate extreme values, we propose an estimation method that recovers the right tail of health expenditure distributions. It extends the popular two-part model to develop a novel three-part model. We apply the proposed method to claims data from one of the biggest German private health insurers. Our findings show that the estimated age gradient in health care spending differs substantially from the standard least squares method.

Are turbulence effects on droplet collision–coalescence a key to understanding observed rain formation in clouds?

Rain formation is a critical factor governing the lifecycle and radiative forcing of clouds and therefore it is a key element of weather and climate. Cloud microphysics-turbulence interactions occur across a wide range of scales and are challenging to represent in atmospheric models with limited resolution. Based on past experiments and idealized numerical simulations, it has been postulated that cloud turbulence accelerates rain formation by enhancing drop collision-coalescence. We provide substantial evidence for significant impacts of turbulence on the evolution of cloud droplet size distributions and rain formation by comparing high-resolution observations of cumulus congestus clouds with state-of-the-art large-eddy simulations coupled with a Lagrangian particle-based microphysics scheme. Turbulent coalescence must be included in the model to accurately represent the observed drop size distributions, especially for drizzle drop sizes at lower heights in the cloud. Turbulence causes earlier rain formation and greater rain accumulation compared to simulations with gravitational coalescence only. The observed rain size distribution tail just above cloud base follows a power law scaling that deviates from theoretical scalings considering either a purely gravitation collision kernel or a turbulent kernel neglecting droplet inertial effects, providing additional evidence for turbulent coalescence in clouds. In contrast, large aerosols acting as cloud condensation nuclei ("giant CCN") do not significantly impact rain formation owing to their long timescale to reach equilibrium wet size relative to the lifetime of rising cumulus thermals. Overall, turbulent drop coalescence exerts a dominant influence on rain initiation in warm cumulus clouds, with limited impacts of giant CCN.

Left and right: a tale of two tails of the wealth distribution

Left and right: a tale of two tails of the wealth distribution

Embracing epistemic uncertainty: a risk evaluation method for pollutants in stormwater.

In this study, we show that pollutants of emerging concern are, by nature, prone to the emergence of epistemic uncertainty. We also show that the current uncertainty quantification methods used for pollutant modelling rely almost exclusively on parameter uncertainty, which is not adequate to tackle epistemic uncertainty affecting the model structure. We, therefore, suggest a paradigm shift in the current pollutant modelling approaches by adding a term explicitly accounting for epistemic uncertainties. In a proof-of-concept, we use this approach to investigate the impact of epistemic uncertainty in the fluctuation of pollutants during wet-weather discharge (input information) on the distribution of mass of pollutants (output distributions). We found that the range of variability negatively impacts the tail of output distributions. The fluctuation time, associated with high covariance between discharge and concentration, is a major driver for the output distributions. Adapting to different levels of epistemic uncertainty, our approach helps to identify critical unknown information in the fluctuation of pollutant concentration. Such information can be used in a risk management context and to design smart monitoring campaigns.

Pan-tumor circulating tumor DNA testing in the community oncology setting.

e14556 Background: Circulating tumor DNA (ctDNA) technology allows for monitoring the genomic profile of cancer over time non-invasively, and in a way that is more representative of existing tumor clones than tissue sampling. The clinical and research applications of this new technology are numerous, and they are potentiated through serial ctDNA testing of individual patients. It is important to identify and characterize real-world data sets that will permit the development of both ctDNA’s clinical and research applications. Methods: This is an observational study of breast, non-small cell lung cancer (NSCLC), colorectal and prostate cancer patients who had ctDNA testing in The US Oncology Network from August 2014 to August 2023. All unique ctDNA reports per patient were collected, along with the report dates and patient characteristics. Rate of change in number of unique reports per month through time was evaluated as well as frequencies of serial testing and median time lapse between diagnosis and date of first ctDNA test by cancer type. Results: Number of new unique ctDNA testing reports per month increased for The US Oncology Network from 2014 through 2018 at a rate of 0.86 (95% CI [0.70, 1.02]). From 2018 to 2023 this rate increased to 5.87 (95% CI [5.16, 6.59]) new reports per month. The total number of unique patients with one of the 4 cancer indications and at least one ctDNA test during the study time period was 10,852. The cancer indication with greatest number of patients with ctDNA testing is NSCLC with 5,503 patients, followed by breast with 2,408, and colorectal and prostate with 1,544 and 1,397 respectively. Black patients were equally represented across all 4 cancers at rates of 13.6% for colorectal, 12.7% for breast, and 12% for NSCLC and prostate. Patients over 65 made up 66% for prostate and NSCLC but only 38% and 32% for colorectal and breast respectively. Most patients presented with a single ctDNA test (n = 9482), but an important proportion presented with 2 serial ctDNA tests or more (n = 1370, 12.6%).The median time between either cancer diagnosis date or date of first visit in The Network and the date of the first ctDNA test report was 39 days for NSCLC, 1,690 days for breast, 377 days for colorectal and 772 days for prostate, and presented with a skewed, long right tailed distribution. Median times were similar for Black and White patients across all cancers. Conclusions: The number of patients with ctDNA testing data available is increasing through time without evidence of plateauing. Although the total number of patients represented is still a very small fraction of the total patient population, the rate of increase in number of new reports through time is equivalent to an exponential increase in the total number of reports. 12.6% of patients presented with two tests or more, potentially allowing for pre and post treatment assessments. There’s an opportunity to better align cancer research and cancer treatment paradigms with this new technology.