Data Depth Function Research Articles

Data depth functions for non-standard data by use of formal concept analysis

In this article we introduce a notion of depth functions for data types that are not given in standard statistical data formats. We focus on data that cannot be represented by one specific data structure, such as normed vector spaces. This covers a wide range of different data types, which we refer to as non-standard data. Depth functions have been studied intensively for normed vector spaces. However, a discussion of depth functions for non-standard data is lacking. In this article, we address this gap by using formal concept analysis to obtain a unified data representation. Building on this representation, we then define depth functions for non-standard data. Furthermore, we provide a systematic basis by introducing structural properties using the data representation provided by formal concept analysis. Finally, we embed the generalised Tukey depth into our concept of data depth and analyse it using the introduced structural properties. Thus, this article presents the mathematical formalisation of centrality and outlyingness for non-standard data and increases the number of spaces in which centrality can be discussed. In particular, we provide a basis for defining further depth functions and statistical inference methods for non-standard data.

Outlier Detection of Functional Data Using Reproducing Kernel Hilbert Space

The problem of finding the pattern that deviates from other observation is termed as outlier. The detection of outlier is getting importance in research area nowadays due to the reason that the technique has been used in various mission critical applications such as military, health care, fault recovery, and many. The analysis of functional data and its depth function plays a crucial role in statistical model for detecting outlier. The depth values alone not enough for finding outliers, since all the low depth values not be an outlier. The main problem of using classical model is that it cannot cop up with the high dimensionality of the data This paper proposed a novel technique based on Reproducing Kernel Hilbert Space curve (RKHS) for detecting outliers in functional data. The proposed RKHS model is based on a special Hilbert space curve associated with a kernel so that it reproduces each function in the space to enhance the performance of data depth function. The proposed method uses distance weighted discrimination classification that avoids overfitting the model and provides better generalizability in high dimensions. The kernel depths perform better performances for detection of outlier in a number of artificial and real data sets.

The GLD-plot: a depth-based graphical tool to investigate unimodality of directional data

A graphical tool for investigating the unimodality of hyperspherical data is proposed. It is based on the notion of statistical data depth function for directional data. A data depth extends the univariate concept of rank and leads to a natural centre-outward ordering of the sample points. First, a local version of distance-based depths for directional data is introduced. Then, the orderings induced by a local depth and its corresponding global version are compared by means of a two-dimensionalscatterplot, i.e., the GLD-plot (global-local depth plot). The graph can be easily visualized and interpreted. The proposal is evaluated on simulated and real data examples.

Multivariate process control charts based on the Lp depth

AbstractEven if large historical dataset could be available for monitoring key quality features of a process via multivariate control charts, previous knowledge may not be enough to reliably identify or adopt a unique model for all the variables. When no specific parametric model turns out to be appropriate, some alternative solutions should be adopted and exploiting non‐parametric methods to build a control chart appears a reasonable choice. Among the possible non‐parametric statistical techniques, data depth functions are gaining a growing interest in multivariate quality control. Within the literature, several notions of depth are effective for this purpose, even in the case of deviation from the normality assumption. However, the use of the Lp depth for constructing non‐parametric multivariate control charts has been surprisingly neglected so far. Hence, the goal of this work is to investigate the behavior the Lp depth in the statistical process control and to compare its performances to those of the Mahalanobis depth, which is often adopted to build depth‐based control charts.

Robust multivariate estimation based on statistical depth filters

In the classical contamination models, such as the gross-error (Huber and Tukey contamination model or case-wise contamination), observations are considered as the units to be identified as outliers or not. This model is very useful when the number of considered variables is moderately small. Alqallaf et al. (Ann Stat 37(1):311–331, 2009) show the limits of this approach for a larger number of variables and introduced the independent contamination model (cell-wise contamination) where now the cells are the units to be identified as outliers or not. One approach to deal, at the same time, with both type of contamination is filter out the contaminated cells from the data set and then apply a robust procedure able to handle case-wise outliers and missing values. Here, we develop a general framework to build filters in any dimension based on statistical data depth functions. We show that previous approaches, e.g., Agostinelli et al. (TEST 24(3):441–461, 2015b) and Leung et al. (Comput Stat Data Anal 111:59–76, 2017), are special cases. We illustrate our method by using the half-space depth.

Depth-based support vector classifiers to detect data nests of rare events

The aim of this project is to combine data depth with support vector machines (SVM) for binary classification. To this end, we introduce data depth functions and SVM and discuss why a combination of the two is assumed to work better in some cases than using SVM alone. For two classes X and Y , one investigates whether an individual data point should be assigned to one of these classes. In this context, our focus lies on the detection of rare events, which are structured in data nests: class X contains much more data points than class Y and Y has less dispersion than X. This form of classification problem is akin to finding the proverbial needle in a haystack. Data structures like these are important in churn prediction analyses which will serve as a motivation for possible applications. Beyond the analytical investigations, comprehensive simulation studies will also be carried out.

A depth-based nearest neighbor algorithm for high-dimensional data classification

Nearest neighbor algorithms like k-nearest neighbors (kNN) are fundamental supervised learning techniques to classify a query instance based on class labels of its neighbors. However, quite often, huge volumes of datasets are not fully labeled and the unknown probability distribution of the instances may be uneven. Moreover, kNN suffers from challenges like curse of dimensionality, setting the optimal number of neighbors, and scalability for high-dimensional data. To overcome these challenges, we propose an improvised approach of classification via depth representation of subspace clusters formed from high-dimensional data. We offer a consistent and principled approach to dynamically choose the nearest neighbors for classification of a query point by i) identifying structures and distributions of data; ii) extracting relevant features, and iii) deriving an optimum value of k depending on the structure of data by representing data using data depth function. We propose an improvised classification algorithm using a depth-based representation of clusters, to improve performance in terms of execution time and accuracy. Experimentation on real-world datasets reveals that proposed approach is at least two orders of magnitude faster for high-dimensional dataset and is at least as accurate as traditional kNN.

Robust mean-variance portfolio through the weighted $$L^{p}$$ depth function

Portfolios constructed by the classical mean-variance model are very sensitive to outliers. We propose the use of a non-parametric estimation method based on statistical data depth functions. Specifically, we exploit the notion of the weighted $$L^{p}$$ depth function to obtain robust estimates of the mean and covariance matrix of the asset returns. This approach has the advantage to be independent of parametric assumptions, and less sensitive to changes in the asset return distribution than traditional techniques. The proposed procedure is evaluated and compared with standard and other robust techniques through simulated and real data. Results indicate effective improvements of the proposed method in terms of out-of-sample performance.

Intrinsic Data Depth for Hermitian Positive Definite Matrices

Nondegenerate covariance, correlation, and spectral density matrices are necessarily symmetric or Hermitian and positive definite. This article develops statistical data depths for collections of Hermitian positive definite matrices by exploiting the geometric structure of the space as a Riemannian manifold. The depth functions allow one to naturally characterize most central or outlying matrices, but also provide a practical framework for inference in the context of samples of positive definite matrices. First, the desired properties of an intrinsic data depth function acting on the space of Hermitian positive definite matrices are presented. Second, we propose two pointwise and integrated data depth functions that satisfy each of these requirements and investigate several robustness and efficiency aspects. As an application, we construct depth-based confidence regions for the intrinsic mean of a sample of positive definite matrices, which is applied to the exploratory analysis of a collection of covariance matrices in a multicenter clinical trial. Supplementary materials and an accompanying R-package are available online.

Non‐convex penalized multitask regression using data depth‐based penalties

We propose a new class of non‐convex penalties based on data depth functions for multitask sparse penalized regression. These penalties quantify the relative position of rows of the coefficient matrix from a fixed distribution centred at the origin. We derive the theoretical properties of an approximate one‐step sparse estimator of the coefficient matrix using local linear approximation of the penalty function and provide an algorithm for its computation. For the orthogonal design and independent responses, the resulting thresholding rule enjoys near‐minimax optimal risk performance, similar to the adaptive lasso (Zou, H (2006), ‘The adaptive lasso and its oracle properties’, Journal of the American Statistical Association, 101, 1418–1429). A simulation study and real data analysis demonstrate its effectiveness compared with some of the present methods that provide sparse solutions in multitask regression. Copyright © 2018 John Wiley & Sons, Ltd.

Uniform convergence rates for halfspace depth

Data depth functions are a generalization of one-dimensional order statistics and medians to real spaces of dimension greater than one; in particular, a data depth function quantifies the centrality of a point with respect to a data set or a probability distribution. One of the most commonly studied data depth functions is halfspace depth. Halfspace depth is of interest to computational geometers because it is highly geometric, and it is of interest to statisticians because it shares many desirable theoretical properties with the one-dimensional median. It is known that as the sample size increases, the halfspace depth for a sample converges to the halfspace depth for the underlying distribution, almost surely. In this paper, we use the geometry and structure of halfspace depth to reduce a high-dimensional problem into many one-dimensional problems. This bound requires only mild assumptions on the distribution, and it leads to an improved convergence rate when the underlying distribution decays exponentially, i.e., the probability that a sample point has magnitude at least R is O(exp(−λR2/2)). We also provide examples and show that our bounds are tight.

Nonparametric catchment clustering using the data depth function

ABSTRACTThe clustering of catchments is important for prediction in ungauged basins, model parameterization and watershed development and management. The aim of this study is to explore a new measure of similarity among catchments, using a data depth function and comparing it with catchment clustering indices based on flow and physical characteristics. A cluster analysis was performed for each similarity measure using the affinity propagation clustering algorithm. We evaluated the similarity measure based on depth–depth plots (DD-plots) as a basis for transferring parameter sets of a hydrological model between catchments. A case study was developed with 21 catchments in a diverse New Zealand region. Results show that clustering based on the depth–depth measure is dissimilar to clustering on catchment characteristics, flow, or flow indices. A hydrological model was calibrated for the 21 catchments and the transferability of model parameters among similar catchments was tested within and between clusters defined by each clustering method. The mean model performance for parameters transferred within a group always outperformed those from outside the group. The DD-plot based method was found to produce the best in-group performance and second-highest difference between in-group and out-group performance.EDITOR D. Koutsoyiannis; ASSOCIATE EDITOR A. Viglione

Analysis of macroseismic fields using statistical data depth functions: considerations leading to attenuation probabilistic modelling

Modelling seismic attenuation is one of the most critical points in the hazard assessment process. In this article we consider the spatial distribution of the effects caused by an earthquake as expressed by the values of the macroseismic intensity recorded at various locations surrounding the epicentre. Considering the ordinal nature of the intensity, a way to show its decay with distance is to draw curves—isoseismal lines—on maps, which bound points of intensity not smaller than a fixed value. These lines usually take the form of closed and nested curves around the epicentre, with highly different shapes because of the effects of ground conditions and of complexities in rupture propagation. Forecasting seismic attenuation of future earthquakes requires stochastic modelling of the decay on the basis of a common spatial pattern. The aim of this study is to consider a statistical methodology that identifies a general shape, if it exists, for isoseismal lines of a set of macroseismic fields. Data depth is a general nonparametric method for analysis of probability distributions and datasets. It has arisen as a statistical method to order points of a multivariate space, e.g., Euclidean space \({\mathbb {R}}^{p}\), \(p \ge 1\), according to the centrality with respect to a distribution or a given data cloud. Recently, this method has been extended to the ordering of functions and trajectories. In our case, for a fixed intensity decay \(\varDelta I\), we build a set of convex hulls that enclose the sites of felt intensity \(I_s \ge I_0 -\varDelta I\), one for each macroseismic field of a set of earthquakes that are considered as similar from the attenuation point of view. By applying data depth functions to this functional dataset, it is possible to identify the most central curve, i.e., the attenuation pattern, and to consider other properties like variability, outlyingness, and possible clustering of such curves. Results are shown for earthquakes that occurred on the Central Po Plain in May 2012, and on the eastern flank of Mt. Etna since 1865.

Training of Artificial Neural Networks Using Information-Rich Data

Artificial Neural Networks (ANNs) are classified as a data-driven technique, which implies that their learning improves as more and more training data are presented. This observation is based on the premise that a longer time series of training samples will contain more events of different types, and hence, the generalization ability of the ANN will improve. However, a longer time series need not necessarily contain more information. If there is considerable repetition of the same type of information, the ANN may not become “wiser”, and one may be just wasting computational effort and time. This study assumes that there are segments in a long time series that contain a large quantum of information. The reason behind this assumption is that the information contained in any hydrological series is not uniformly distributed, and it may be cyclic in nature. If an ANN is trained using these segments rather than the whole series, the training would be the same or better based on the information contained in the series. A pre-processing can be used to select information-rich data for training. However, most of the conventional pre-processing methods do not perform well due to large variation in magnitude, scale and many zeros in the data series. Therefore, it is not very easy to identify these information-rich segments in long time series with large variation in magnitude and many zeros. In this study, the data depth function was used as a tool for the identification of critical (information) segments in a time series, which does not depend on large variation in magnitude, scale or the presence of many zeros in data. Data from two gauging sites were used to compare the performance of ANN trained on the whole data set and just the data from critical events. Selection of data for critical events was done by two methods, using the depth function (identification of critical events (ICE) algorithm) and using random selection. Inter-comparison of the performance of the ANNs trained using the complete data sets and the pruned data sets shows that the ANN trained using the data from critical events, i.e., information-rich data (whose length could be one third to half of the series), gave similar results as the ANN trained using the complete data set. However, if the data set is pruned randomly, the performance of the ANN degrades significantly. The concept of this paper may be very useful for training data-driven models where the training time series is incomplete.

Use of the data depth function to differentiate between case of interpolation and extrapolation in hydrological model prediction

Summary Hydrological models are subject to significant sources of uncertainty including input data, model structure and parameter uncertainty. A key requirement for an operational flow forecasting model is therefore to give accurate estimates of model uncertainty. This estimate is often presented in terms of confidence bounds. The quality and quantity of observed rainfall and flow data available for calibration has a great influence on the identification of hydrological model parameters, and hence the model error distribution and width of the confidence bounds. The information contained in the observed time series is not uniformly distributed, and may not represent all types of behaviour or activation of flow pathways that could occur in the catchment. A model calibrated with data from a given time period could therefore perform well or poorly when evaluated over a new time period, depending on the information content and variability of the calibration data, in relation to the validation period. Our hypothesis is that we can improve the estimate of hydrological predictive uncertainty, based on our knowledge of the range of data available for calibration. If the characteristics of the validation data are similar in information content and variability to those in the calibration period, we term this an “interpolation case”, and expect the model errors during calibration to be similar to those in validation. Otherwise, it is an “extrapolation case”, where we may expect model errors to be greater. In this study, we developed an algorithm to differentiate cases of ‘interpolation’ versus ‘extrapolation’ in the prediction time period. The algorithm is based on the concept of ‘data depth’, i.e. the location of new data in relation to the convex hull of the calibration data set. Using a case study, we calculated uncertainty bounds for the predictive time period using methods with/without differentiation of interpolation and extrapolation cases. The performance of the resulting confidence bounds in accurately representing model error was evaluated using both visual inspection for specific events, and rank histograms and range statistics for the simulated model error quantiles. We show that the proposed algorithm enables us to give differentiated predictive uncertainty bounds which represent model error in more realistic ways.

Improving the calibration strategy of the physically-based model WaSiM-ETH using critical events

The use of a physically-based hydrological model for streamflow forecasting is limited by the complexity in the model structure and the data requirements for model calibration. The calibration of such models is a difficult task, and running a complex model for a single simulation can take up to several days, depending on the simulation period and model complexity. The information contained in a time series is not uniformly distributed. Therefore, if we can find the critical events that are important for identification of model parameters, we can facilitate the calibration process. The aim of this study is to test the applicability of the Identification of Critical Events (ICE) algorithm for physically-based models and to test whether ICE algorithm-based calibration depends on any optimization algorithm. The ICE algorithm, which uses the data depth function, was used herein to identify the critical events from a time series. Low depth in multivariate data is an unusual combination and this concept was used to identify the critical events on which the model was then calibrated. The concept is demonstrated by applying the physically-based hydrological model WaSiM-ETH on the Rems catchment, Germany. The model was calibrated on the whole available data, and on critical events selected by the ICE algorithm. In both calibration cases, three different optimization algorithms, shuffled complex evolution (SCE-UA), parameter estimation (PEST) and robust parameter estimation (ROPE), were used. It was found that, for all the optimization algorithms, calibration using only critical events gave very similar performance to that using the whole time series. Hence, the ICE algorithm-based calibration is suitable for physically-based models; it does not depend much on the kind of optimization algorithm. These findings may be useful for calibrating physically-based models on much fewer data. Editor D. Koutsoyiannis; Associate editor A. Montanari Citation Singh, S.K., Liang, J.Y., and Bárdossy, A., 2012. Improving calibration strategy of physically-based model WaSiM-ETH using critical events. Hydrological Sciences Journal, 57 (8), 1487–1505.

Lens data depth and median

We define the lens depth (LD) function LD(t; F) of a vector t∈R d with respect to a distribution function F to be the probability that t is contained in a random hyper-lens formed by the intersection of two closed balls centred at two i.i.d observations from F. We show that LD is a statistical depth function and explore its properties, including affine invariance, symmetry, maximality at the centre and monotonicity. We define the sample LD and investigate its uniform consistency, asymptotic normality and computational complexity in high-dimensional settings. We define the lens median (LM), a multivariate analogue of the univariate median, as the point where the LD is maximised. The sample LM is the vector that is covered by the most number of hyper-lenses formed between any two sample observations. We state its asymptotic consistency and normality and examine its breakdown point and relative efficiency. The sample LM is robust and efficient for estimating the centre of a unimodal distribution. A comparison of LD and LM to existing data depth functions and medians in terms of computational complexity, robustness, efficiency and breakdown point is presented.

Second-order accuracy of depth-based bootstrap confidence regions

We consider the problem of setting bootstrap confidence regions for multivariate parameters based on data depth functions. We prove, under mild regularity conditions, that depth-based bootstrap confidence regions are second-order accurate in the sense that their coverage error is of order n − 1 , given a random sample of size n . The results hold in general for depth functions of types A and D, which cover as special cases the Tukey depth, the majority depth, and the simplicial depth. A simulation study is also provided to investigate empirically the bootstrap confidence regions constructed using these three depth functions.