Empirical Distribution Of Data Research Articles

Differential analysis of Operating System indicators for anomaly detection in dependable systems: An experimental study

Dependable complex systems often operate under variable and non-stationary conditions, which requires efficient and extensive monitoring and error detection solutions. Among the many, the paper focuses on anomaly detection techniques, which monitor the evolution of some specific indicators through time to identify anomalies, i.e. deviations from the expected operational behavior. The timely identification of anomalies in dependable, fault tolerant systems allows to timely detect errors in the services and react appropriately. In this paper, we investigate the possibility to monitor the evolution of indicators through time using the random walk model on indicators belonging to Operating Systems, specifically in our study the Linux Red Hat EL5. The approach is based on the experimental evaluation of a large set of heterogeneous indicators, which are acquired under different operating conditions, both in terms of workload and faultload, on an air traffic management target system. The statistical analysis is based on a best-fitting approach aiming to minimize the integral distance between the empirical data distribution and some reference distributions. The outcomes of the analysis show that the idea of adopting a random walk model for the development of an anomaly detection monitor for critical systems that operates at Operating System level is promising. Moreover, standard distributions such as Laplace and Cauchy, rather than Normal, should be used for setting up the thresholds of the monitor. Further studies that involve a new application, a different Operating System and a new layer (an Application Server) will allow verifying the generalization of the approach to other fault tolerant systems, monitored layers and set of indicators.

Statistical analysis of bistatic and monostatic sea clutter

In this work, we analyze the sea clutter data collected simultaneously by the bistatic and monostatic nodes of an S-band netted radar system under different out-of-plane geometries. The empirical distribution of sea clutter data are fitted to some known heavy-tailed distributions, and the sea spikes statistics are examined. Our analyses show that spikiness is higher for small bistatic angles and that bistatic data are sometimes less spiky than monostatic ones only for horizontal polarization.

Evolving Scale-Free Networks by Poisson Process: Modeling and Degree Distribution.

Since the great mathematician Leonhard Euler initiated the study of graph theory, the network has been one of the most significant research subject in multidisciplinary. In recent years, the proposition of the small-world and scale-free properties of complex networks in statistical physics made the network science intriguing again for many researchers. One of the challenges of the network science is to propose rational models for complex networks. In this paper, in order to reveal the influence of the vertex generating mechanism of complex networks, we propose three novel models based on the homogeneous Poisson, nonhomogeneous Poisson and birth death process, respectively, which can be regarded as typical scale-free networks and utilized to simulate practical networks. The degree distribution and exponent are analyzed and explained in mathematics by different approaches. In the simulation, we display the modeling process, the degree distribution of empirical data by statistical methods, and reliability of proposed networks, results show our models follow the features of typical complex networks. Finally, some future challenges for complex systems are discussed.

Time-Efficient Algorithms for Robust Estimators of Location, Scale, Symmetry, and Tail heaviness

The analysis of the empirical distribution of univariate data often includes the computation of location, scale, skewness, and tail-heaviness measures, which are estimates of specific parameters of the underlying population distribution. Several measures are available, but they differ by Gaussian efficiency, robustness regarding outliers, and meaning in the case of asymmetric distributions. In this article, we briefly compare, for each type of parameter (location, scale, skewness, and tail heaviness), the “classical” estimator based on (centered) moments of the empirical distribution, an estimator based on specific quantiles of the distribution, and an estimator based on pairwise comparisons of the observations. This last one always performs better than the other estimators, particularly in terms of robustness, but it requires a heavy computation time of an order of n2. Fortunately, as explained in Croux and Rousseeuw (1992, Computational Statistics 1: 411–428), the algorithm of Johnson and Mizoguchi (1978, SIAM Journal of Scientific Computing 7: 147–153) allows one to substantially reduce the computation time to an order of n log n and, hence, allows the use of robust estimators based on pairwise comparisons, even in very large datasets. This has motivated us to program this algorithm for Stata. In this article, we describe the algorithm and the associated commands. We also illustrate the computation of these robust estimators by involving them in a normality test of Jarque–Bera form (Jarque and Bera 1980, Economics Letters 6: 255–259; Brys, Hubert, and Struyf, 2008, Computational Statistics 23: 429–442) using real data.

Confidence interval of nonlinear regression of restoration time of network terminal devices

Building confidence interval of the nonlinear regression equation and estimating the restoration time of devices play an important role in the practical tasks of terminal network control. Restoration time is non-Gaussian random variable, which depends on the distance between the service center and the terminal network device. For more reliable estimation of restoration time, it is necessary to have the confidence interval of its non-linear regression.In the case of Gaussian random variable, it is possible to apply linear regression equation and build the confidence interval for it by traditional method using the Student's t-distribution. This method does not take into account many features of the empirical data distribution, for example, its asymmetry. In the case of non-Gaussian random variable, it is difficult to build the confidence interval of the nonlinear regression equation without the assumption of normality.Applying linearizing transformations is reduced to obtaining a linear regression model from the original non-linear by replacing variables and coefficients. This substitution leads to the model simplification and data loss, associated with nonlinearity.Using the normalizing transformations allows to proceed to linear regression of the normalized data, build the confidence interval for it by the traditional method, and finally, by applying the relevant transformation, pass to non-linear regression and its confidence interval. Johnson transformation is used as normalizing transformation because it provides the best results as compared to other known transformations.

A Nash equilibrium simulation model for the competitiveness evaluation of the auction based day ahead electricity market

This paper presents a simulation model based on the Nash equilibrium notion for the auction based day ahead electricity generation market. The presented model enhances a previous formalism proposed in the related literature by employing empirical data distributions of the market clearing price as registered by the market authority (e.g. the Independent System Operator). The model is effective when power suppliers with different generation capacities are considered, differently from the starting model that unrealistically assumes equal capacities. The proposed approach aims at evaluating the electricity market competitiveness with regard to the bidder strategies in order to prevent their anticompetitive actions. The framework is applied to a real data set regarding the Italian electricity market to enlighten its effectiveness in different scenarios, varying the number and capacity of participating bidders. The model can be employed as a basis for a decision support tool both for market participants (to define their optimal bidding strategy) and regulators (to avoid collusive strategies).

On the approximation of empirical data for service system simulations

The difficulties associated with parameter estimation for phase-type approximations of empirical data distributions in queue modelling are well known. While significant progress has been achieved in improving such approximations, difficulties in parameter estimation still limit the extent to which queue modelling is applied in practice. This paper presents a simplified technique for approximating empirical data in service system simulations, based on the specialised Cox phase-type distribution. When utilised in simulation modelling of a service system, the specialised Cox distribution is shown to provide improved approximations to the combined waiting and service time distribution without the need for complex parameter estimation techniques. This approach should enable much greater flexibility in the application of queue modelling to service systems.

Development of Method for Three-Point Data Estimation and SVR-QSAR Model to Screen Anti Cancer Leads

Present work deals with generation of virtual samples as mathematical modeling of empirical data on the basis of empirical data. The generated samples were used for development of QSAR model. The method deals with extrapolation of sample vector in such a manner that there is conservation of the empirical data distribution. The data distribution has been judged with statistical parameters. The method was implemented with anticancer activity of Gossypol acetic acid against BCL2 target for colorectal cancer. Considering the virtual samples only for model development, model training showed a regression coefficient for leave one out cross validation as 0.996 with 66 virtual samples, and a regression coefficient with external test set data (51 samples) as 0.993. External test set data which were never used in the virtual sample generation showed predicted regression coefficient value of >0.61. On the basis of QSAR model, nine compounds were suggested as anti-BCL2 active compounds. The suggested compounds were further validated by docking study with Gossypol acetic acid and 'Tetrahydroisoquinoline amide substituted phenyl pyrazole' cocrystallized with chimeric BCL2-XL (PDBID: 2W3L) protein.

Optimization of the number of components in the mixed model using multi-criteria decision-making

The distributions of empirical data are often complex. Such complexity cannot be sufficiently addressed by the individual theoretical statistical distribution function. Furthermore, the selection of the distribution function becomes more complicated when the empirical data present a multi-peak feature. In such a case, the multiple testing criteria and the mixed model must be considered during the selection of an appropriate distribution function. Aiming at this vague challenge, the present paper proposes a novel method for establishing a mixed model that can describe accurately the distribution characteristics of empirical data. Apart from combining the Akike and Bayesian information criteria to define the feasible solutions of the mixed model, this study also utilizes the root mean squared deviation, coefficient of determination, Kolmogorov–Smirnov test statistic, average deviation in cumulative distribution function, and average deviation in probability distribution function as the testing criteria. In addition, a non-linear programming is used to find the weighting factors of each criterion. The multi-criteria decision-making technology is adopted to comprehensively and objectively integrate these testing criteria into a synthetic indicator. Finally, an optimization algorithm is proposed to determine the optimal number of components in the mixed model. The illustrated results of the simulated data and measured signals confirm that this approach can estimate precisely the number of components as well as establish a highly accurate mixed model.

Bayesian mixture modeling approach to account for heterogeneity in speed data

Speed is one of the most important parameters describing the condition of the traffic flow. Many analytical models related to traffic flow either produce speed as a performance measure, or use speed to determine other measures such as travel time, delay, and the level of service. Mathematical models or distributions used to describe speed characteristics are very useful, especially when they are utilized in the context of simulation and theoretical derivations. Traditionally, normal, log–normal and composite distributions have been the usual mathematical distributions to characterize speed data. These traditional distributions, however, often fail to produce an adequate goodness-of-fit when the empirical distribution of speed data exhibits bimodality (or multimodality), skewness, or excess kurtosis (peakness). This often occurs when the speed data are generated from several different sub-populations, for example, mixed traffic flow conditions or mixed vehicle compositions. The traditional modeling approach also lacks the ability to explain the underlying factors that lead to different speed distribution curves. The objective of this paper is to explore the applicability of the finite mixture of normal (Gaussian) distributions to capture the heterogeneity in vehicle speed data, and thereby explaining the aforementioned special characteristics. For the parameter estimation, Bayesian estimation method via Markov Chain Monte Carlo (MCMC) sampling is adopted. The field data collected on IH-35 in Texas is used to evaluate the proposed models. The results of this study show that the finite mixture of normal distributions can very effectively describe the heterogeneous speed data, and provide richer information usually not available from the traditional models. The finite mixture modeling produces an excellent fit to the multimodal speed distribution curve. Moreover, the causes of different speed distributions can be identified through investigating the components.

Financial Data and the Skewed Generalized T Distribution

This paper develops a skewed extension of the generalized t (GT) distribution, introduced by McDonald and Newey (1988). In particular, the paper derives the mathematical moments and other properties of the distribution and assesses its ability to fit the empirical distribution of several financial series characterized by skewness and excess kurtosis. In all cases the skewed GT provides an excellent fit to the empirical distribution of data.

Extending Lotkaian informetrics

The continuous version of the Lotka distribution, more generally referred to outside of informetrics as the Pareto distribution, has long enjoyed a central position in the theoretical development of informetrics despite several reported drawbacks in modelling empirical data distributions, most particularly that the inverse power form seems mainly to be evident only in the upper tails. We give a number of published examples graphically illustrating this shortcoming. In seeking to overcome this, we here draw attention to an intuitively reasonable generalization of the Pareto distribution, namely the Pareto type II distribution, of which we consider two versions. We describe its basic properties and some statistical features together with concentration aspects and argue that, at least in qualitative terms, it is better able to describe many observed informetric phenomena over the full range of the distribution. Suggestions for further investigations, including truncated and time-dependent versions, are also given.

Using Bayesian networks for bankruptcy prediction: Some methodological issues

This study provides operational guidance for building naïve Bayes Bayesian network (BN) models for bankruptcy prediction. First, we suggest a heuristic method that guides the selection of bankruptcy predictors. Based on the correlations and partial correlations among variables, the method aims at eliminating redundant and less relevant variables. A naïve Bayes model is developed using the proposed heuristic method and is found to perform well based on a 10-fold validation analysis. The developed naïve Bayes model consists of eight first-order variables, six of which are continuous. We also provide guidance on building a cascaded model by selecting second-order variables to compensate for missing values of first-order variables. Second, we analyze whether the number of states into which the six continuous variables are discretized has an impact on the model’s performance. Our results show that the model’s performance is the best when the number of states for discretization is either two or three. Starting from four states, the performance starts to deteriorate, probably due to over-fitting. Finally, we experiment whether modeling continuous variables with continuous distributions instead of discretizing them can improve the model’s performance. Our finding suggests that this is not true. One possible reason is that continuous distributions tested by the study do not represent well the underlying distributions of empirical data. Finally, the results of this study could also be applicable to business decision-making contexts other than bankruptcy prediction.

Extended Correlation

What is the correlation between two variables? Traditional answers offer summary assessments such as Pearson’s r and regression coefficients. But new computing techniques make it possible to construct conceptually simple hypotheses that describe the full joint distribution of two variables, making it possible to “mine” the correlation for information that was previously unused. This article begins with evidence of systematic anomalies in the empirical joint distribution of height-weight data and follows with a hypothesis that explains these anomalies in terms of a theoretical joint distribution relative to a linear equation. The hypothesis has serious consequences because even in traditional examples, while it offers an improved fit to the data, its estimates of the linear center do not correspond to traditional least squares estimates of the linear relation for the same two variables.

Background and threshold: critical comparison of methods of determination

Different procedures to identify data outliers in geochemical data are reviewed and tested. The calculation of [mean±2 standard deviation (sdev)] to estimate threshold values dividing background data from anomalies, still used almost 50 years after its introduction, delivers arbitrary estimates. The boxplot, [median±2 median absolute deviation (MAD)] and empirical cumulative distribution functions are better suited for assisting in the estimation of threshold values and the range of background data. However, all of these can lead to different estimates of threshold. Graphical inspection of the empirical data distribution using a variety of different tools from exploratory data analysis is thus essential prior to estimating threshold values or defining background. There is no good reason to continue to use the [mean±2 sdev] rule, originally proposed as a ‘filter’ to identify approximately 2½% of the data at each extreme for further inspection at a time when computers to do the drudgery of numerical operations were not widely available and no other practical methods existed. Graphical inspection using statistical and geographical displays to isolate sets of background data is far better suited for estimating the range of background variation and thresholds, action levels (e.g., maximum admissible concentrations—MAC values) or clean-up goals in environmental legislation.

Quantification of Variability and Uncertainty for Air Toxic Emission Inventories with Censored Emission Factor Data

Probabilistic emission inventories were developed for urban air toxic emissions of benzene, formaldehyde, chromium, and arsenic for the example of Houston. Variability and uncertainty in emission factors were quantified for 71-97% of total emissions, depending upon the pollutant and data availability. Parametric distributions for interunit variability were fit using maximum likelihood estimation (MLE), and uncertainty in mean emission factors was estimated using parametric bootstrap simulation. For data sets containing one or more nondetected values, empirical bootstrap simulation was used to randomly sample detection limits for nondetected values and observations for sample values, and parametric distributions for variability were fit using MLE estimators for censored data. The goodness-of-fit for censored data was evaluated by comparison of cumulative distributions of bootstrap confidence intervals and empirical data. The emission inventory 95% uncertainty ranges are as small as -25% to +42% for chromium to as large as -75% to +224% for arsenic with correlated surrogates. Uncertainty was dominated by only a few source categories. Recommendations are made for future improvements to the analysis.

An out-of-equilibrium model of the distributions of wealth

The distribution of wealth among the members of a society is herein assumed to result from two fundamental mechanisms: trade and investment. An empirical distribution of wealth shows an abrupt change between the low–medium range, that may be fitted by a non-monotonic function with an exponential-like tail such as a gamma distribution, and the high wealth range, that is well fitted by a Pareto or inverse power-law function. We demonstrate that an appropriate trade-investment model, depending on three adjustable parameters associated with the total wealth of a society, a social differentiation among agents, and economic volatility referred to as investment, can successfully reproduce the distribution of empirical wealth data in the low, medium and high ranges. Finally, we provide an economic interpretation of the mechanisms in the model and, in particular, we discuss the difference between classical and neoclassical theories regarding the concepts of value and price. We consider the importance that out-of-equilibrium trade transactions, where the prices differ from values, have in real economic societies.

MODELING FINANCIAL SERIES DISTRIBUTIONS: A VERSATILE DATA FITTING APPROACH

The empirical distribution of common stock returns is a subject of interest to many researchers, as it often determines the validity of theoretical models proposed in the economics and finance studies. This paper brings to the attention the availability of two flexible systems of distributions for fitting data: the Johnson system of distributions and the Tadikamalla–Johnson system of distributions. We explore the feasibility of fitting the empirical distributions of several financial series to these two systems of distributions. Both systems of distributions are highly flexible and capable of accommodating all possible skewness and kurtosis values. The probability density function and the cumulative distribution function take on simple closed forms and appropriate transformations of the data lead to normal/logistic distributions. In addition, the parameter estimation procedures are easy to implement. When the results are compared with those of other data fitting models, in all cases tested, the proposed distributions provide a good fit to the empirical distribution of data.

Parametric and non-parametric statistical analysis of DT-MRI data

In this work parametric and non-parametric statistical methods are proposed to analyze Diffusion Tensor Magnetic Resonance Imaging (DT-MRI) data. A Multivariate Normal Distribution is proposed as a parametric statistical model of diffusion tensor data when magnitude MR images contain no artifacts other than Johnson noise. We test this model using Monte Carlo (MC) simulations of DT-MRI experiments. The non-parametric approach proposed here is an implementation of bootstrap methodology that we call the DT-MRI bootstrap. It is used to estimate an empirical probability distribution of experimental DT-MRI data, and to perform hypothesis tests on them. The DT-MRI bootstrap is also used to obtain various statistics of DT-MRI parameters within a single voxel, and within a region of interest (ROI); we also use the bootstrap to study the intrinsic variability of these parameters in the ROI, independent of background noise. We evaluate the DT-MRI bootstrap using MC simulations and apply it to DT-MRI data acquired on human brain in vivo, and on a phantom with uniform diffusion properties.

An application of exploratory data analysis (EDA) as a robust non-parametric technique for geochemical mapping in a semi-arid climate

Exploratory data analysis (EDA) techniques based on the boxplot and robust-class selection were applied to the analysis of single-element stream sediment data in the Collo area (N-E Algeria). The area is characterised by many factors that affect data variability: variation of lithology, combined existence of permanent and ephemeral streams, flash floods, rugged terrain, and climate. The boxplot proved to be very useful in capturing the empirical data distribution, the skewness, and in defining outliers. No data transformation was needed prior to the analysis of single element distributions (Cr, Pb, Zn, Cu, As, and Fe) as is the case in classical statistics. Geochemical mapping of these elements was based on resistant class selection as defined by the boxplot. Results showed the close spatial correlation of outlier data for Cr with a plagioclase-lherzolite intrusion and known chromite pods. The geochemical maps of Pb, Zn, Cu and As concentrations showed an association of these elements coincident with known base-metal sulphides and arsenopyrite mineralisation, delineating a northeastern anomaly spreading well over known mineralisation. The choice of the robust classes based on the boxplot also showed the close spatial distribution of Fe with known hematite and magnetite mineralization. EDA techniques proved to be very useful in delineating known mineralisation in the Collo area where stream sediment data is subject to variability owing to many factors (geologic, physiographic and climatic). EDA proved to be a simple and very useful tool in analysing single-element geochemical data. EDA could be used in similar terrains as an alternative to the classical methods where a normality precondition is needed prior to analysis and where class selection may be affected by the presence of “wild” data.