Sample Skewness Research Articles

Flood frequency analysis using generalized distributions and entropy-based model selection method

Flood frequency analysis (FFA) is used for estimating the return period of a design flood. Fundamental to FFA is the selection of a frequency distribution fitted to the data set, where an inappropriate choice of a distribution can lead to significant error and bias in the design flood estimate. The usual criteria for selecting a distribution are Akaike Information Criterion (AIC), Bayesian Information Criterion (BIC), and Root Mean Square Error (RMSE), although these criteria have limitations. This paper performed FFA using three main steps. First, five generalized distributions were considered as candidate distributions for FFA; second, the principle of maximum entropy (POME) and the method of maximum likelihood (MLE) were used for parameter estimation; and third a new model selection criterion based on the value of entropy was proposed to choose the best-fitted distribution. Monte Carlo simulation was carried out to test this entropy-based method for four sample sizes equal to 50, 200, 1000, and 10000, and simulation was repeated 1000 times for each sample size. Then, the probability that a given method identified the correct distribution was determined. The probabilities affected by the sample size, skewness, and shape of the probability density function (PDF) were assessed. Using Qing River basin, China, as a case study, the design flood values were calculated and compared. Results of simulation showed that the proposed method was better than other methods for the following cases: (a) the sample size of data set X is small; (b) the skewness coefficient CS>0; and (c) the shape of PDFs is bell-shaped.

Outliers Detection in Skewed Distributions: Split Sample Skewness Based Boxplot

Outliers Detection in Skewed Distributions: Split Sample Skewness Based Boxplot

Split sample skewness

The shape of a statistical distribution of data is important in such diverse areas as descriptive analysis, risk analysis, portfolio optimization and strategic decision making. If it is known (or believed) that the probability density function of a random variable is not symmetric, the question of its skewness becomes important. Various methods of assessing skewness have been formulated, but none are totally satisfactory. The classical measurement of skewness is based on higher moments of the random variable about its mean. However, it is well known that those measurements are sensitive to extreme outliers. The implication of the sensitivity is that mean-based metrics of skewness are inefficient, especially in small or medium size sample data. Those mean-based metrics are known as Pearson skewness, Quartile skewness and Octile skewness. All have been devised to try to accommodate for the presence of outliers. However, those test statistics are demonstrably inefficient in the presence of outliers. That inefficiency motivates the approach in this paper; the development of an efficient and more robust skewness metric we call Split Sample Skewness, hereafter referred to as SSS. In this context, efficiency means that the SSS metric requires fewer sample observations than the less efficient metrics cited above to achieve the same level of statistical robustness. The name reflects a methodology that partitions the sample into two subgroups at the median. This paper displays the findings of multiple simulation studies adducing evidence bearing on the efficiency of split ample skewness relative to other measures of skewness.

Testing normality of data on a multivariate grid

We propose a significance test to determine if data on a regular d-dimensional grid can be assumed to be a realization of Gaussian process. By accounting for the spatial dependence of the observations, we derive statistics analogous to sample skewness and kurtosis. We show that the sum of squares of these two statistics converges to a chi-square distribution with two degrees of freedom. This leads to a readily applicable test. We examine two variants of the test, which are specified by two ways the spatial dependence is estimated. We provide a careful theoretical analysis, which justifies the validity of the test for a broad class of stationary random fields. A simulation study compares several implementations. While some implementations perform slightly better than others, all of them exhibit very good size control and high power, even in relatively small samples. An application to a comprehensive data set of sea surface temperatures further illustrates the usefulness of the test.

Nearly unbiased estimation of sample skewness

In this paper we examine the finite sample bias of sample skewness estimator for financial returns. We show that the bias of conventional sample skewness comes from two sources: the covariance between past return and future volatility, known as the leverage effect, and the covariance between past volatility and future return, commonly referred to as the volatility feedback effect. We derive explicit expressions for this bias and propose a nearly unbiased skewness estimator under mild assumptions. Our simulation study shows that the proposed estimator leads to almost unbiased skewness estimates with a sightly elevated mean squared error, and can reduce the bias of the skewness coefficient estimates by 40%. In our empirical application, we find that bias-corrected average skewness can better predict future market returns comparing to the case without bias-correction. This leads to an improved performance of skewness-based portfolios in terms of Sharpe ratio, certainty equivalence and transaction cost.

The geometric mean?

The sample geometric mean (SGM) introduced by Cauchy in 1821, is a measure of central tendency with many applications in the natural and social sciences including environmental monitoring, scientometrics, nuclear medicine, infometrics, economics, finance, ecology, surface and groundwater hydrology, geoscience, geomechanics, machine learning, chemical engineering, poverty and human development, to name a few. Remarkably, it was not until 2013 that a theoretical definition of the population geometric mean (GM) was introduced. Analytic expressions for the GM are derived for many common probability distributions, including: lognormal, Gamma, exponential, uniform, Chi-square, F, Beta, Weibull, Power law, Pareto, generalized Pareto and Rayleigh. Many previous applications of SGM assumed lognormal data, though investigators were unaware that for that case, the GM is the median and SGM is a maximum likelihood estimator of the median. Unlike other measures of central tendency such as the mean, median, and mode, the GM lacks a clear physical interpretation and its estimator SGM exhibits considerable bias and mean square error, which depends significantly on sample size, pd, and skewness. A review of the literature reveals that there is little justification for use of the GM in many applications. Recommendations for future research and application of the GM are provided.

Tests of Normality of Functional Data

SummaryThe paper is concerned with testing normality in samples of curves and error curves estimated from functional regression models. We propose a general paradigm based on the application of multivariate normality tests to vectors of functional principal components scores. We examine finite sample performance of a number of such tests and select the best performing tests. We apply them to several extensively used functional data sets and determine which can be treated as normal, possibly after a suitable transformation. We also offer practical guidance on software implementations of all tests we study and develop large sample justification for tests based on sample skewness and kurtosis of functional principal component scores.

Large Underestimation of Intraspecific Trait Variation and Its Improvements.

Intraspecific trait variation (ITV) is common feature of natural communities and has gained increasing attention due to its significant ecological effects on community dynamics and ecosystem functioning. However, the estimation of ITV per se has yet to receive much attention, despite the need for accurate ITV estimation for trait-based ecological inferences. It remains unclear if, and to what extent, current estimations of ITV are biased. The most common method used to quantify ITV is the coefficient of variation (CV), which is dimensionless and can therefore be compared across traits, species, and studies. Here, we asked which CV estimator and data normalization method are optimal for quantifying ITV, and further identified the minimum sample size required for ±5% accuracy assuming a completely random sample scheme. To these ends, we compared the performance of four existing CV estimators, together with new simple composite estimators, across different data normalizations, and sample sizes using both a simulated and empirical trait datasets from local to regional scales. Our results consistently showed that the most commonly used ITV estimator (CV 1= σsample/μsample), often underestimated ITV—in some cases by nearly 50%—and that underestimation varies largely among traits and species. The extent of this bias depends on the sample size, skewness and kurtosis of the trait value distribution. The bias in ITV can be substantially reduced by using log-transforming trait data and alternative CV estimators that take into consideration the above dependencies. We find that the CV4 estimator, also known as Bao's CV estimator, combined with log data normalization, exhibits the lowest bias and can reach ±5% accuracy with sample sizes greater than 20 for almost all examined traits and species. These results demonstrated that many previous ITV measurements may be substantially underestimated and, further, that these underestimations are not equal among species and traits even using the same sample size. These problems can be largely solved by log-transforming trait data first and then using the Bao's CV to quantify ITV. Together, our findings facilitate a more accurate understanding of ITV in community structures and dynamics, and may also benefit studies in other research areas that depend on accurate estimation of CV.

Chemical surveillance in freshwaters: small sample sizes underestimate true pollutant loads and fail to detect environmental quality standard exceedances

BackgroundChemical surveillance in surface waters is crucial to identify potential threats to the health of freshwater ecosystems. Usually, the concentrations of pollutants are highly variable over the course of the year and often result in non-normally distributed data sets. Therefore, the European Water Framework Directive recommends measuring, e.g. priority substances at least 12 times a year to achieve an acceptable accuracy level for the estimation of the true mean annual loads. However, in Europe priority substances are often measured much less frequently. In this context, the aim of the present study was to analyze how sample size, temporal variability and skewness of the data sets influence the accuracy of the mean annual load estimation and the assessment of annual average environmental quality standards. For this purpose, sample size simulations using weekly composite samples of benzo(a)pyrene, 4-tert-octylphenol, fluoranthene and di(2-ethylhexyl) phthalate, selected as representatives for priority substances, were carried out.ResultsThe sample size simulations showed two general patterns: the accuracy of the mean annual load estimation increased with increasing sample size and skewness and temporal variability were more apparent in smaller sample sizes. In right-skewed data sets, small sample sizes led, on average, to a systematic underestimation of the true mean annual load whilst in a few cases these led to an overestimation. Although the study was carried out on priority substances, results can be transferable to other pollutants. Furthermore, in small sample sizes a considerable proportion of the simulated means failed to detect annual average environmental quality standard exceedances.ConclusionsThe results of the present study indicate that the usage of small sample sizes is likely to result in an underestimation of the true mean annual pollutant loads in chemical surveillance and scientific research, thus potentially jeopardizing the validity of results. Therefore, it is recommended to avoid the usage of small sample sizes for the determination of mean annual pollutant loads. Furthermore, priority substances should be sampled according to the European Water Framework Directive guidelines at least 12 times/year to improve the assessment of the threat posed by pollutants to freshwater ecosystems in Europe.

HACSim: an R package to estimate intraspecific sample sizes for genetic diversity assessment using haplotype accumulation curves.

Assessing levels of standing genetic variation within species requires a robust sampling for the purpose of accurate specimen identification using molecular techniques such as DNA barcoding; however, statistical estimators for what constitutes a robust sample are currently lacking. Moreover, such estimates are needed because most species are currently represented by only one or a few sequences in existing databases, which can safely be assumed to be undersampled. Unfortunately, sample sizes of 5–10 specimens per species typically seen in DNA barcoding studies are often insufficient to adequately capture within-species genetic diversity. Here, we introduce a novel iterative extrapolation simulation algorithm of haplotype accumulation curves, called HACSim (Haplotype Accumulation Curve Simulator) that can be employed to calculate likely sample sizes needed to observe the full range of DNA barcode haplotype variation that exists for a species. Using uniform haplotype and non-uniform haplotype frequency distributions, the notion of sampling sufficiency (the sample size at which sampling accuracy is maximized and above which no new sampling information is likely to be gained) can be gleaned. HACSim can be employed in two primary ways to estimate specimen sample sizes: (1) to simulate haplotype sampling in hypothetical species, and (2) to simulate haplotype sampling in real species mined from public reference sequence databases like the Barcode of Life Data Systems (BOLD) or GenBank for any genomic marker of interest. While our algorithm is globally convergent, runtime is heavily dependent on initial sample sizes and skewness of the corresponding haplotype frequency distribution.

A Covert Digital Communication System Using Skewed α-Stable Distributions for Internet of Things

Information security is a key issue involved in the utilization of different technological innovations, such as the Internet of Things (IoT). This study proposes a new covert communication scheme in which the transmitted waveform modulated by the covert bit is statistically similar to the background noise to achieve information security. The impulsive noise in many communication fields can be well modeled with a symmetrical α -stable distribution. Thus, the proposed scheme exploits skewed α -stable distributions to conduct the covert bit. The polarity transition of the skewness parameter within the first and second halves of a bit period represents a covert bit. The resultant signal exhibits a near-symmetrical behavior similar to ambient interference to achieve concealment. The sample skewness estimator based on the signed fractional lower order moment is used to determine the binary message at the receiver. Then, the expression of the skewness estimator and its probability density function are derived. With the aid of the hard decision, the theoretical expression of the bit error rate of the proposed communication system is obtained. The proposed scheme works at the physical layer with the advantages of simple structure, easy implementation, and extreme suitability with low-cost, resource-limited IoT devices exhibiting low-rate transmission.

Kullback–Leibler divergence and sample skewness for pathological voice quality assessment

This paper proposes new features aiming to improve the performance of an automatic voice pathology detection system. The features are designed precisely in terms of voice pathologies effects upon the speech signal. The system is intended to deliver high accuracy with a low number of parameters. Kullback–Leibler divergence (KLD) applied to consecutive frames of the speech signal provides a measure of voice instability. In this work, the KLD is applied to frame’s histogram and a modified form of its spectrum named higher amplitude suppression spectrum (HASS). The H-KLD (histogram KLD) and the HASS-KLD are two of the three features presently approached. An additional feature that provides the level of damping of the voice pitch period waveform is proposed, the short-term sample skewness of the signal. The H-KLD, the HASS-KLD, and the sample skewness are features employed along with mel-frequency cepstral coefficients (MFCC) in a voice pathology detection system. The system is composed of a Gaussian mixture models (GMM) classifier and two generalized extreme value (GEV) distribution classifiers. They are fused by means of a Gaussian naïve Bayes classifier. A standard subset of the Massachusetts Eye and Ear Infirmary (MEEI) voice disorders database is adopted for evaluating the system. The obtained global success rate of 99.55% shows that the proposed features are suitable for pathological voice quality assessment.

Vascular flow on doppler sonography may not be a valid characteristic to distinguish colloid nodules from papillary thyroid carcinoma even when accounting for nodular size.

The purpose of this study was to test the hypothesis that there is no significant difference in vascular flow patterns between cytopathologically-proven colloid nodules and papillary thyroid carcinoma (PTC) even when adjusting for nodule size. Doppler vascular flow patterns in 200 colloid nodules and 166 nodules with PTC were retrospective reviewed independently by 2 neuroradiologists blinded to the cytopathological results. Absence of vascular flow, perinodular and/or intranodular flow, and diffuse vascular flow were recorded. The vascular flow patterns were compared without (Fisher exact test) and with (Kruskal-Wallis test) an adjustment for nodular size. Using the most common flow pattern as the reference group, multiple logistic regression was used to compare the flow patterns. Sample skewness was calculated to determine degree of symmetry of the size distribution for each vascular flow category. No significant difference was found in the tested vascular flow patterns between colloid nodules and PTC both without and with an adjustment for nodular size (P>0.05). Intranodular flow only was the largest group (n=111/366) and used as the reference for multiple logistic regression. No significant difference was noted between the vascular flow patterns (P>0.05). Sample skewness showed that nodules were generally smaller in size with outliers of larger size on the opposite end of the spectrum. Independent of nodule size the absence or presence of vascular flow is not significantly different between colloid nodules and PTC. Therefore, vascular flow may not be useful in distinguishing between colloid nodules and PTC.

Risk Indicators for Hearing Impairment in Neonates

To identify relation between prevalence of hearing impairment and risk factors at the time of birth in high risk neonates. A year-long study is performed at HBT Medical College and Dr. R. N. Cooper Municipal General Hospital, Vile Parle (west), Mumbai. The Study comprised of 478 high risk babies born between March 2016 and February 2017 and admitted in NICU of the hospital. Out of these 68 babies were either transferred to higher centre or took discharge against medical advice. A total of 410 babies completed the study. Additional 21 eligible babies died in NICU during the study. A Prospective Observational Study is used to analyse the findings of the study. Skew analysis of spread of prevalence across risk factors is used to establish higher prevalence of hearing impairment in high risk neonates. Cramer's Moment Coefficient of Skewness and Corresponding Test Statistics are used for studying the skewness. Out of 410 babies screened during the study 5 babies were diagnosed with profound SNHL. The risk factors associated with babies is plotted against frequency of babies with Profound SNHL. The plot is found to be un-uniformly distributed and shows inclination towards higher prevalence with high risk babies. Sample skewness (G1) value - 0.81 indicates moderately negative skewed sample. This means the number of babies with profound SNHL found by 3 stage screening is moderately concentrated at high risk weightage.

Modified Lilliefors goodness-of-fit test for normality

The first aim of the paper is to put into practice the -corrected Lilliefors goodness-of-fit test for normality (LF). This modification consists in varying a formula of calculating the empirical distribution function (EDF). Values of constants in the formula depend on values of sample skewness and excess kurtosis, which is recommended in order to increase the power of the LF test. Critical values are obtained with the Monte Carlo method for sample sizes and at a significance level The power of several normality tests for a wide collection of alternative distributions is calculated. Alternative distributions are divided into 12 groups according to their skewness and excess kurtosis. The second aim is to propose a similarity measure between the normal distribution and an alternative distribution. The third aim is to propose two new alternative distributions created in order to obtain the desired values of skewness and excess kurtosis. The fourth aim is to calculate the power of tests for new alternative distributions. The paper shows that values of constants in the formula of calculating the EDF influences the power of the LF test. The performance of the new proposal through the analysis of real data sets is illustrated.

Robust measures of skewness and kurtosis for macroeconomic and financial time series

ABSTRACTThe sample skewness and kurtosis of macroeconomic and financial time series are routinely scrutinized in the early stages of model-building and are often the central topic of studies in economics and finance. Notwithstanding the availability of several robust estimators, most scholars in economics rely on method-of-moments estimation that is known to be very sensitive to outliers. We carry out an extensive Monte Carlo analysis to evaluate the bias and root mean squared error of 12 different estimators of skewness and kurtosis. We consider nine statistical distributions that approximate the range of data generating processes of many macroeconomic and financial time series. Both in independently and identically distributed samples and in data generating processes featuring serial correlation L-moments and trimmed L-moments estimators are particularly resistant to outliers and deliver improvements over standard as well as alternative robust estimators of skewness and kurtosis. The application to 128 macroeconomic and financial time series sourced from a large, monthly frequency, database (i.e. the FRED-MD of McCracken and Ng, 2016) confirms the findings of the simulation study.

Profiting from correlations: Adjusted estimators for categorical data

AbstractTo take sample biases and skewness in the observations into account, practitioners frequently weight their observations according to some marginal distribution. The present paper demonstrates that such weighting can indeed improve the estimation. Studying contingency tables, estimators for marginal distributions are proposed under the assumption that another marginal is known. It is shown that the weighted estimators have a strictly smaller asymptotic variance whenever the two marginals are correlated. The finite sample performance is illustrated in a simulation study. As an application to traffic accident data the method allows for correcting a well‐known bias in the observed injury severity distribution.

Robust Measures of Skewness and Kurtosis for Macroeconomic and Financial Time Series

The sample skewness and kurtosis of macroeconomic and financial time series are routinely scrutinized in the early stages of model-building and are often the central topic of studies in economics and finance. Notwithstanding the availability of several robust estimators, most scholars in economics rely on method-of-moments estimation that is known to be very sensitive to outliers. We carry out an extensive Monte Carlo analysis to compare the bias and root mean squared error of twelve different estimators of skewness and kurtosis. We consider nine statistical distributions that approximate the range of data generating processes of many macroeconomic and financial time series. Both in independently and identically distributed samples and in data generating processes featuring serial correlation L-moments and trimmed L-moments estimators are particularly resistant to outliers and deliver the lowest root mean squared error. The application to 128 macroeconomic and financial time series sourced from a large, monthly frequency, database (i.e. the FRED-MD of McCracken and Ng, 2016) confirms the findings of the simulation study.

Development of Hartigan’s Dip Statistic with Bimodality Coefficient to Assess Multimodality of Distributions

In general, although some random variables such as wind speed, temperature, and load are known to have multimodal distributions, input or output random variables are considered to follow unimodal distributions without assessing the unimodality or multimodality of distributions from samples. In uncertainty analysis, estimating unimodal distribution as multimodal distribution or vice versa can lead to erroneous analysis results. Thus, whether a distribution is unimodal or multimodal must be assessed before the estimation of distributions. In this paper, the bimodality coefficient (BC) and Hartigan’s dip statistic (HDS), which are representative methods for assessing multimodality, are introduced and compared. Then, a combined HDS with BC method is proposed. The proposed method has the advantages of both BC and HDS by using the skewness and kurtosis of samples as well as the dip statistic through a link function between the BC values in BC and significance level in HDS. To verify the performance of the proposed method, statistical simulation tests were conducted to evaluate the multimodality for various unimodal, bimodal, and trimodal models. The implementation of the proposed method to real engineering data is shown through case studies. The results demonstrate that the proposed method is more accurate, robust, and reliable than the BC and original HDS alone.

The Alpha stable distribution in ocean ambient noise modelling

In view of the non-Gaussian of ocean ambient noise, the  stable distribution is applied to the statistical modelling. Firstly, the one-to-one correspondence between the four parameters of stable distribution and the sample mean, variance, skewness and kurtosis are established according to physical meaning. Then, numerical simulations are conducted to analyze the suitability of stable distribution for non-Gaussian ambient noise. In the case of white noise interference, noise is divided into Gaussian state, leptokurtic, and platykurtic separately. The parameters of stable distribution are estimated by the sample quantile and characteristic function method jointly. The simulation results show that, in the Gaussian state,  stable distribution is equivalent to normal distribution. As for leptokurtic distribution, stable distribution is much better than normal distribution, indicating absolute predominance in impulse-like data modeling. But it is not adaptive for low kurtosis state because its characteristic exponent can’t be bigger than two. Finally, the result is verified by ambient noise collected in three environmental conditions, such as quiet ambient noise, airgun interference noise and ship noise. In all three cases,  stable distribution shows good adaptability and accuracy, especially for the airgun dataset it is far superior to normal distribution.