Exploratory Data Analysis Techniques Research Articles

Tail density estimation for exploratory data analysis using kernel methods

ABSTRACTIt is often critical to accurately model the upper tail behaviour of a random process. Nonparametric density estimation methods are commonly implemented as exploratory data analysis techniques for this purpose and can avoid model specification biases implied by using parametric estimators. In particular, kernel-based estimators place minimal assumptions on the data, and provide improved visualisation over scatterplots and histograms. However kernel density estimators can perform poorly when estimating tail behaviour above a threshold, and can over-emphasise bumps in the density for heavy tailed data. We develop a transformation kernel density estimator which is able to handle heavy tailed and bounded data, and is robust to threshold choice. We derive closed form expressions for its asymptotic bias and variance, which demonstrate its good performance in the tail region. Finite sample performance is illustrated in numerical studies, and in an expanded analysis of the performance of global climate models.

Variabilidade sazonal e interanual da chuva para o noroeste do estado do Espírito Santo, Brazil

Rain is one of the climatic elements that exert greater influence on environmental conditions. The frequency of rainfall distribution follows annual trends, and can therefore be based on historical data. This study aimed to evaluate the frequency and intensity of rainfall in the city of Colatina, State of Espirito Santo, from 2000 to 2015, using statistical techniques of exploratory data analysis. Based on these results, it was found that the total annual rainfall average for the series analyzed was 1029.9 mm, revealing the existence of a Z negative trend of -4.98 of annual rainfall by the Mann-Kendall test. It was possible to identify, with the Pettitt test, abrupt change of annual precipitation in relation to the average of the studied period from the year 2013. The water balance resulted in ten months of soil water deficit with a total annual of 350.9 mm and the surplus occurred in November and December with 27.8 mm. It is concluded that the continuation of this downward trend in rainfall totals in the city could result in impacts on economic activities, environment and society.

Spatial dynamics of cultural diversity in the Netherlands

In this paper, we analyse the spatial dimension of changing ethnic diversity at the neighbourhood level. Drawing from recent work on income convergence, we characterise the evolution of population diversity in the Netherlands over space. Our analysis is structured over three dimensions, which allow us to find clear spatial patterns in how cultural diversity changes at the neighbourhood level. Globally, we use directional statistics to visualise techniques of exploratory data analysis, finding a clear trend towards ‘spatially integrated change’: a situation where the trajectory of ethnic change in a neighbourhood is closely related to that in adjacent neighbourhoods. When we zoom into the local level, a visualisation of recent measures of local concordance allows us to document a high degree of spatial heterogeneity in how the overall change is distributed over space. Finally, to further explore the nature and characteristics of neighbourhoods that experience the largest amount of change, we develop a spatial, multilevel model. Our results show that the largest cities, as well as those at the boundaries with Belgium and Germany, with the most diverse neighbourhoods, have large clusters of stable neighbourhood diversity over time, while concentrations of high dynamic areas are nearby these largest cities. The analysis shows that neighbourhood diversity spatially ‘spills over’, gradually expanding outside traditionally diverse areas.

The spatial distribution and relationship of tourist flow in Turkey

Tourism is considered as back bone of Turkish economy with its greater share in GDP than many other economic sectors of the country. Present research is sought to analyse the spatial association and distribution of foreign and domestic tourist inflows in 81 provinces of Turkey through exploratory spatial data analysis (ESDA) technique. ESDA is one of very useful Geographic Information System (GIS) based spatial statistics techniques to analyse spatial patterns, identification of hotspots and visualization of spatial association among variables. Analysis of global Moran’s I statistics results into a positive value that indicates presence of spatial autocorrelation among neighbouring provinces with high(low) number of both foreign and domestic tourist arrivals during years 2000-2016. This depicts positive and negative autocorrelation of places with geographical similarities and dissimilarities over the space. Moreover, Moran’s significance maps help indicating hotspot areas of high and low tourist arrivals where high clusters are found in western coastal areas and lower clusters are associated to eastern inland areas. Furthermore, Moran’s scatterplot analysis highlights the regional disparities within the country in terms of tourism development. This polarized spatial pattern of tourism is associated with differences in economic development and resource allocation between western and eastern provinces. This study is significant for policy makers by providing insights into spatial distribution of tourist flows for better resource allocation and management of hotspot areas. Besides, it also helps private sector and tour operators for economic utilization of tourist clusters and hotspots.

Spatial Analysis of Environmental Regulations: Evidence from Chinese Provinces

This research focuses on the spatial distribution and spatial correlation of environmental regulations (ERS) with exploratory spatial data analysis technique (ESDA). The panel data of 31 provinces in China during the period of 2005-2015 is used. Finds show that the most intensive ERS region is located around Beijing-Tianjin-Hebei region. The values of global Moran’s I vary from 0.1104 to 0.2145, indicating that ERS in China is not distributed randomly, but has spatial agglomeration effects. The High–High agglomeration type is mainly distributed in north China.

Chemical characterization of atmospheric particulate matter in Friuli Venezia Giulia (NE Italy) by exploratory data analysis with multisite and multivariate approach.

The chemical composition of atmospheric particulate (PM10) in the Friuli Venezia Giulia (FVG) region (NE Italy) has been characterized for the first time with the help of exploratory data analysis (EDA) techniques (uni-, bi-, and multivariated, i.e., principal components analysis), molecular and elemental diagnostic ratios, and seasonal trends. Despite that the available analytical data was limited to the parameters routinely analyzed on PM10 by ARPA FVG (11 elements and 16 PAH congeners), the large number of samples and of measured chemical parameters, together with the applied techniques of data analysis, allowed us to extract useful latent information from the dataset, resulting in a greater knowledge of both regional and local features. Specifically, we succeeded in matching data patterns to the known pollution sources of some sampling stations, both industrial (two secondary fusion steelworks and one coke oven) and urban (traffic and domestic heating), and in defining the mainly urban or mainly industrial feature of some questionable sampling stations. This is of paramount importance to check for possible industrial inputs in urban stations, allowing policymakers to implement the most appropriate response.

A Survey on Multidimensional Scaling

This survey presents multidimensional scaling (MDS) methods and their applications in real world. MDS is an exploratory and multivariate data analysis technique becoming more and more popular. MDS is one of the multivariate data analysis techniques, which tries to represent the higher dimensional data into lower space. The input data for MDS analysis is measured by the dissimilarity or similarity of the objects under observation. Once the MDS technique is applied to the measured dissimilarity or similarity, MDS results in a spatial map. In the spatial map, the dissimilar objects are far apart while objects which are similar are placed close to each other. In this survey article, MDS is described in comprehensive fashion by explaining the basic notions of classical MDS and how MDS can be helpful to analyze the multidimensional data. Later on, various special models based on MDS are described in a more mathematical way followed by comparisons of various MDS techniques.

The Steaming Mug

The case is meant to be used in an introductory course on Data Visualization or Analytics in either a business school or any other context where students have had some exposure to management topics. This case primarily focuses on introducing students to the basic philosophy and techniques of exploratory data analysis (EDA). The Steaming Mug, a US based hot beverage chain is provided as a context for engaging in a hands-on exploratory data analysis exercise to understand the strengths and weaknesses of the company's operations and performance. The learning from exploring this case will help students understand that many different insights can be drawn from the given data. The students will understand that there is no “one correct path” for this analysis. The values of getting multiple groups to present their findings is that students realise that there are many other angles that could have been explored, than the path they had gone down.

On the Use of Redundancy Analysis to Study the Property Crime in Poland

Redundancy analysis (RDA) is a canonical form of principal components analysis (PCA) and is one of, so‑called, linear ordination techniques. The goal of ordination is to represent objects and response variables relationships as faithfully as possible in a low‑dimensional space. Redundancy analysis is also a technique of exploratory data analysis. Graphical presentation of the results using the ordination biplots or triplots can facilitate the analysis of the relationship between the variation in the set of the response variables and the variation of the explanatory variables. In the paper, redundancy analysis was applied to assess the relationships between the selected socio‑economic factors and the intensity of the crime against property in Poland.

Optimization of computer aided detection systems: An evolutionary approach

Computer Aided Diagnosis (CAD) systems are designed to aid the radiologist in interpreting medical images. They are usually based on lesion detection and segmentation algorithms whose performance depends on a large number of parameters. While time consuming and sub-optimal, parameter values are most often selected through manual search strategies. Genetic or evolutionary algorithms (GA) are effective optimization methods that mimic biological evolution. Genetic algorithms have been shown to efficiently manage complex search spaces, and can be applied to all kinds of objective functions, including discontinuous, nondifferentiable, or highly nonlinear ones. In this study, we have adopted an evolutionary approach to the problem of parameter optimization. We show that the genetic algorithm is able to effectively converge to a better solution than manual optimization on a case study for digital breast tomosynthesis CAD. Parameter optimization was framed as a constrained optimization problem, where the function to be maximized was defined as the weighted sum of sensitivity, false positive rate and segmentation accuracy. A modified Dice coefficient was defined to assess the segmentation quality of individual lesions. Finally, all viable solutions evaluated by the GA were studied by means of exploratory data analysis techniques, such as association rules, to gain useful insights on the strength of the influence of each parameter on overall algorithm performance. We showed that this combination was able to identify multiple ranges of viable solutions with good segmentation accuracy.

Performance Analysis of Partition and Evolutionary Clustering Methods on Various Cluster Validation Criteria

Large quantity of data has been accumulating tremendously due to digitalization. But the accumulated data are not converted into useful patterns. This gap is conquered by using exploratory data analysis techniques. Clustering is one of the vital technologies in exploratory data analysis. It is a methodology to arrange data objects as per their characteristics. Traditional clustering approaches, namely leader, K-means, ISODATA and evolutionary-based approaches like genetic algorithm, particle swarm optimization, social group optimization methods, are also implemented on benchmark data set. Evolutionary-based clustering methods are derived from the existing hard clustering methods for finding optimal results. Performance analysis of the above clustering techniques should be validated through different cluster validation methods. The performance analysis reveals evolutionary clustering methods convergence rate is better than partition clustering methods. ISODATA performs better in various aspects on large data. In this work analyzed performance of hard and evolutionary clustering methods on execution time, internal cluster validity criteria.

Eficiência Produtiva Regional da Agricultura Brasileira: uma análise de fronteira estocástica

Resumo: O objetivo deste estudo é analisar a eficiência produtiva da agricultura em escala microrregional usando técnicas de estimação de fronteiras de produção estocásticas e análise exploratória de dados espaciais. As estimações foram feitas para as lavouras permanentes e temporárias separadamente com dados do Censo Agropecuário 2006. Os resultados da equação de produção mostram que os fatores produtivos área de terra e trabalho representam ganhos de produtividade para essas lavouras ligados aos retornos crescentes de escala e à intensidade tecnológica. Os resultados também indicam que a relação entre eficiência e especialização é positiva nas lavouras permanentes e negativa nas lavouras temporárias. Fatores geográficos relacionados com precipitações, temperatura e biomas se mostraram mais relevantes para explicar os diferenciais regionais de eficiência das lavouras temporárias. A análise espacial aplicada à lavoura permanente identificou clusters de alta eficiência nas regiões Nordeste e Sul e um núcleo de baixa eficiência na região Norte. Para as lavouras temporárias, destacam-se poucos clusters de alta eficiência situados nas regiões Nordeste e Sudeste e áreas extensas de baixa eficiência situadas predominantemente no Nordeste e Norte do Brasil.

Yoonhwan

Yoonhwan

Texture analysis in liver of common carp (Cyprinus carpio) sub-chronically exposed to perfluorooctanoic acid

An operator-neutral, objective method was implemented to comparatively assess liver pathology in 30 specimens of common carp (Cyprinus carpio): 20 after experimental flow-through exposure to two perfluorooctanoic acid (PFOA) dosages (10 fish exposed to 200ngl−1 and 10 fish exposed to 2mgl−1) for 56days and 10 unexposed (negative control). The method relies on texture analysis as a complementary approach to traditional histopathology and chemical dosage analysis performed previously on the same experimental material. Texture features data were analyzed by means of Redundancy Analysis (RDA), Linear Discriminant Analysis (LDA) and Canonical Variates Analysis (CVA). LDA resulted in the correct classification of 80% of cases (24 out of 30 cases) with a sensitivity of 83.3% and a specificity of 83.3. In particular, four male samples from the low dosage group (200ngl−1) were misclassified as unexposed fish and two female samples from the unexposed group were misclassified as low dosage exposed. Nevertheless, PFOA liver chemical dosage analysis results were the same both in unexposed and in low dosage group fish, all below the limit of detection. No sample from the high dosage group (2mgl−1) has ever been misclassified. Interestingly, texture features correlated with the PFOA concentrations detected in the liver of each sampled fish. In the present study the technique of texture analysis was combined with techniques of multivariate exploratory data analysis (RDA, LDA/CVA). This approach resulted in a robust, sufficiently sensitive and specific means to study PFOA-induced liver pathology. The new method can discriminate between unexposed and two PFOA exposed groups with better confidence and in a more affordable way, compared to chemical quantification of liver PFOA. The texture features correlated well with liver PFOA concentrations and objectively quantified degenerative liver morphology. In conclusion the overall approach may be a suitable candidate as a reliable and broad-ranging method for biomarker analysis of exposure and effect.

Application of feature selection methods for automated clustering analysis: a review on synthetic datasets

The effective modelling of high-dimensional data with hundreds to thousands of features remains a challenging task in the field of machine learning. This process is a manually intensive task and requires skilled data scientists to apply exploratory data analysis techniques and statistical methods in pre-processing datasets for meaningful analysis with machine learning methods. However, the massive growth of data has brought about the need for fully automated data analysis methods. One of the key challenges is the accurate selection of a set of relevant features, which can be buried in high-dimensional data along with irrelevant noisy features, by choosing a subset of the complete set of input features that predicts the output with higher accuracy comparable to the performance of the complete input set. Kohonen’s self-organising neural network map has been utilised in various ways for this task, such as with the weighted self-organising map (WSOM) approach and this method is reviewed for its efficacy. The study demonstrates that the WSOM approach can result in different results on different runs on a given dataset due to the inappropriate use of the steepest descent optimisation method to minimise the weighted SOM’s cost function. An alternative feature weighting approach based on analysis of the SOM after training is presented; the proposed approach allows the SOM to converge before analysing the input relevance, unlike the WSOM that aims to apply weighting to the inputs during the training which distorts the SOM’s cost function, resulting in multiple local minimums meaning the SOM does not consistently converge to the same state. We demonstrate the superiority of the proposed method over the WSOM and a standard SOM in feature selection with improved clustering analysis.

Novel Text Analysis for Investigating Personality: Identifying the Dark Lady in Shakespeare’s Sonnets

With the rise of social media, understanding a person’s sentiment, their mood and anxiety levels, has become just as important to law enforcement agencies as identifying online anonymous blog posts that aim to radicalize, promote terrorism, or criminal activity. Many of these mood and identity techniques rely on basic statistical correlations, word counts, collocated word groups, or keyword density. We claim that an alternative technique that uses word semantics reflecting personality or characteristics of self can provide a more accurate profile of a person. To test this we analyse Shakespeare’s Sonnets, a collection of poems believed to contain a Dark Lady and two other ‘voices’. We use an exploratory combinatorial data analysis technique called seriation in combination with RPAS, a multi-faceted text analysis approach that draws on a writer’s personality, or self, to visualize the 154 sonnets and to investigate whether it is possible to identify subtle characteristics within a person’s writing style from small texts when an author’s identity is known. We find that RPAS, not only clusters the Dark Lady and other sonnets, but also has the potential to discriminate subtle shifts in personality from texts as small as 90 words.

Determinants of Corporate Philanthropy: A Case of Karachi Stock Exchange

Corporate Philanthropy (CP) has become increasingly important as a means of filling gaps in the provision of social services using Public-Private partnerships. Very little research exists on determinants and motives of CP in Pakistan. This paper utilizes data from the 2001-2011 Annual Reports of 234 firms listed on the Karachi Stock Exchange (KSE) to analyze the association of CP with different firm specific factors like firm size, firm visibility, labor intensity, firm’s financial position (measured as profitability, leverage, cash and equivalent balance, and dividend payout), and the impact of natural disasters on CP. Because this is the first study on this data set, we have employed exploratory data analysis techniques and carefully consider the impact of outliers and other non-standard features of the data. We have used Panel data analytic techniques as well as the Tobit model to analyze the associations between different firm specific factors and CP. We found that firm size, prior profitability, and natural disaster have significant positive impact on CP. We found no evidence that firms are using philanthropy for image creation. Furthermore, exploratory data analysis shows that there are some clusters of firms, which behave differently. Standard regression analysis techniques cannot deal with such cases that require more sophisticated treatment.

An Alternative Classification Scheme for Uncertain Attribute Mapping

The reality of uncertain data cannot be ignored. Anytime that spatial data are used to assist planning, decision making, or policy generation, it is likely that error or uncertainty in the data will propagate through processing protocols and analytic techniques, potentially leading to biased or incorrect decision making. The ability to directly account for uncertainty in spatial analysis efforts is critically important. This article focuses on addressing data uncertainty in one of the most important and widely used exploratory spatial data analysis (ESDA) techniques—choropleth mapping—and proposes an alternative map classification method for uncertain spatial data. The classification approach maximizes within-class homogeneity under data uncertainty while explicitly integrating spatial characteristics to reduce visual map complexity and to facilitate pattern perception. The method is demonstrated by mapping the 2009 to 2013 American Community Survey estimates of median household income in Salt Lake County, Utah, at the census tract level.

Coordinated development of a coupled social economy and resource environment system: a case study in Henan Province, China

At present, social economic development is undergoing severe stress with far-reaching impacts for the resource environment throughout the world. Based on the concept of coordination, the interaction between the social economic and resource environment can be modeled by a coupling coordination degree model. Using this method, we evaluated the social economic index and resource environment index based on an improved catastrophe progression model. Panel data comprising 21 factors were collected from 2001, 2004, 2007, 2010, and 2013 for 18 cities in Henan, China, and used to analyze developmental progress. The coupling coordination degree was also studied using the exploratory spatial data analysis technique and rescaled range analysis, which explained the patterns in spatial variation and its future trends. The results showed are as follows: (1) Social economic development and resource environment status exhibited an increasing trend. Cities were classified into four categories according to their average evaluation ranking in terms of social economic and the resource environment subsystem. (2) The coupling coordination class changed from slight disorder depression to moderate coordinated development, which reflected the optimization of internal subsystems. (3) An approximately inverted U-shaped curve was obtained for the coupling coordination degree in both the east–west and north–south directions. The overall spatial pattern in Henan was “middle high and low in surrounding areas.” (4) In the future, the extent of coordination is expected to increase continuously. Thus, Henan should focus on green development, accelerate the establishment of a resource-saving and environmentally friendly society, promote sustainable economic development, and facilitate the sustainable utilization of the resource environment.

Prediction of Hexaconazole Concentration in the Top Most Layer of Oil Palm Plantation Soil Using Exploratory Data Analysis (EDA).

Ganoderma boninense is a fungus that can affect oil palm trees and cause a serious disease called the basal stem root (BSR). This disease causes the death of more than 80% of oil palm trees midway through their economic life and hexaconazole is one of the particular fungicides that can control this fungus. Hexaconazole can be applied by the soil drenching method and it will be of interest to know the concentration of the residue in the soil after treatment with respect to time. Hence, a field study was conducted in order to determine the actual concentration of hexaconazole in soil. In the present paper, a new approach that can be used to predict the concentration of pesticides in the soil is proposed. The statistical analysis revealed that the Exploratory Data Analysis (EDA) techniques would be appropriate in this study. The EDA techniques were used to fit a robust resistant model and predict the concentration of the residue in the topmost layer of the soil.