Linear Rank Statistics Research Articles

Learning to rank anomalies: scalar performance criteria and maximization of rank statistics

AbstractThe ability to collect and store ever more massive data, unlabeled in many cases, has been accompanied by the need to process them efficiently in order to extract relevant information and possibly design solutions based on the latter. In various situations, the vast majority of the observations exhibit the same behavior, while a small proportion deviates from it. Detecting these outlier observations (or equivalently defined as anomalies) is now one of the major challenges for machine learning applications (e.g. fraud detection or predictive maintenance). We propose here a novel methodology for outlier/anomaly detection, by learning a scoring function defined on the feature space allowing for ranking the observations by degree of abnormality. The scoring function is built through maximization of an empirical performance criterion taking the form of a (two-sample) linear rank statistic. We show that bipartite ranking algorithms can thus be used to learn nearly optimal scoring function with provable theoretical guarantees. We illustrate our methodology with numerical experiments based on open access online code.

Point process convergence for symmetric functions of high-dimensional random vectors

The convergence of a sequence of point processes with dependent points, defined by a symmetric function of iid high-dimensional random vectors, to a Poisson random measure is proved. This also implies the convergence of the joint distribution of a fixed number of upper order statistics. As applications of the result a generalization of maximum convergence to point process convergence is given for simple linear rank statistics, rank-type U-statistics and the entries of sample covariance matrices.

The multi-aspect tests in the presence of ties

The two-sample problem is one of the most important topics in various fields, such as biomedical experiments and product quality maintenance. The Lepage-type test, which is the sum of squares of standardized linear rank statistics, has often been used in the location-scale shift model. Recently, the Lepage-type test has been applied to the joint location-scale and joint location-scale-shape problems. In this study, the test statistics based on the Euclidean distance and Mahalanobis distance of standardized linear rank statistics are considered in the presence of ties. The moments of these test statistics are calculated by deriving the moment-generating function of the vector of linear rank statistics. Moreover, the gamma approximation based on these moments is compared with the chi-square approximation based on the limiting null distribution. Simulation studies and data examples demonstrate the usefulness of gamma approximation in the case of small sample sizes.

Randomization-based Joint Central Limit Theorem and Efficient Covariate Adjustment in Randomized Block 2 K Factorial Experiments

Randomized block factorial experiments are widely used in industrial engineering, clinical trials, and social science. Researchers often use a linear model and analysis of covariance to analyze experimental results; however, limited studies have addressed the validity and robustness of the resulting inferences because assumptions for a linear model might not be justified by randomization in randomized block factorial experiments. In this article, we establish a new finite population joint central limit theorem for usual (unadjusted) factorial effect estimators in randomized block 2 K factorial experiments. Our theorem is obtained under a randomization-based inference framework, making use of an extension of the vector form of the Wald–Wolfowitz–Hoeffding theorem for a linear rank statistic. It is robust to model misspecification, numbers of blocks, block sizes, and propensity scores across blocks. To improve the estimation and inference efficiency, we propose four covariate adjustment methods. We show that under mild conditions, the resulting covariate-adjusted factorial effect estimators are consistent, jointly asymptotically normal, and generally more efficient than the unadjusted estimator. In addition, we propose Neyman-type conservative estimators for the asymptotic covariances to facilitate valid inferences. Simulation studies and a clinical trial data analysis demonstrate the benefits of the covariate adjustment methods. Supplementary materials for this article are available online.

An R-Estimator in the Errors in Variables Linear Regression Model

This note develops an R estimator of the regression parameters in the errors in variables linear regression model, when the distributions of the vectors of covariates and measurement errors are known. The paper also contains the proof of the asymptotic uniform linearity of a sequence of a simple linear rank statistics based on the residuals of a class of nonlinear parametric regression models where covariates and regression errors are possibly dependent. This result in turn facilitates the proof of the asymptotic normality of the above mentioned R estimator in the errors in variables linear regression model. The Pitman’s asymptotic relative efficiency of this R estimator relative to the bias corrected least squares estimator is shown to increase to infinity as the measurement error variance increases to infinity at some Gaussian errors and covariate distributions.

Distribution-free Phase-I scheme for location, scale and skewness shifts with an application in monitoring customers' waiting time

Phase-I analysis of historical data from a statistical process is a strategic problem in Statistical Process Monitoring and control. Before the establishment of process stability, it is challenging to model historical data. Consequently, a distribution-free approach is a natural choice in Phase-I monitoring. Existing distribution-free Phase-I control charts are suitable for detecting instability in location and scale parameters only and are often insensitive in complex processes involving skewness or shape parameters. A new Phase-I control chart is proposed to identify more general shifts, including location, scale and skewness. The proposed Phase-I scheme is efficient in such a situation. The proposed Phase-I scheme uses subsamples, and the plotting statistic is based on the omnibus multi-sample linear rank statistic corresponding to the location, scale and skewness shifts. The new scheme can identify subsamples that are not in control, and it can also indicate one or more process parameters where a deviation has occurred. The encouraging performance of the proposed scheme is established with a large-scale numerical study based on Monte-Carlo in detecting shifts of various nature in a comprehensive class of situations. An illustration based on monitoring the waiting time data from a customer service centre is given. Some concluding remarks and some future research problems are also offered.

Concentration inequalities for two-sample rank processes with application to bipartite ranking

The ROC curve is the gold standard for measuring the performance of a test/scoring statistic regarding its capacity to discriminate between two statistical populations in a wide variety of applications, ranging from anomaly detection in signal processing to information retrieval, through medical diagnosis. Most practical performance measures used in scoring/ranking applications such as the AUC, the local AUC, the p-norm push, the DCG and others, can be viewed as summaries of the ROC curve. In this paper, the fact that most of these empirical criteria can be expressed as two-sample linear rank statistics is highlighted and concentration inequalities for collections of such random variables, referred to as two-sample rank processes here, are proved, when indexed by VC classes of scoring functions. Based on these nonasymptotic bounds, the generalization capacity of empirical maximizers of a wide class of ranking performance criteria is next investigated from a theoretical perspective. It is also supported by empirical evidence through convincing numerical experiments.

Erratum to “On asymptotic normality of certain linear rank statistics” [Statist. Probab. Lett. 145 (2019) 63–73

In our paper On asymptotic normality of certain linear rank statistics, Statistics & Probability Letters, Volume 145, 2019, Pages 63–73, https://doi.org/10.1016/j.spl.2018.08.016., we have noticed an arithmetic error in the proof of Proposition 3.3. The latter error is easy to correct and the correction strengthens current version of Proposition 3.3. The related Proposition 3.4 remains valid without significant alterations in the proofs. Some minor typos are also fixed.

Assessing the Optimal Cutpoint for Tumor Size in Patients with Lung Cancer Based on Linear Rank Statistics in a Competing Risks Framework.

PurposeIn clinical studies, patients may experience several types of events during follow up under the competing risks (CR) framework. Patients are often classified into low- and high-risk groups based on prognostic factors. We propose a method to determine an optimal cutpoint value for prognostic factors on censored outcomes in the presence of CR.Materials and MethodsWe applied our method to data collected in a study of lung cancer patients. From September 1, 1991 to December 31, 2005, 758 lung cancer patients received tumor removal surgery at Samsung Medical Center in Korea. The proposed statistic converges in distribution to that of the supremum of a standardized Brownian bridge. To overcome the conservativeness of the test based on an approximation of the asymptotic distribution, we also propose a permutation test based on permuted samples.ResultsMost cases considered in our simulation studies showed that the permutation-based test satisfied a significance level of 0.05, while the approximation-based test was very conservative: the powers of the former were larger than those of the latter. The optimal cutpoint value for tumor size (unit: cm) prior to surgery for classifying patients into two groups (low and high risks for relapse) was found to be 1.8, with decent significance reflected as p values less than 0.001.ConclusionThe cutpoint estimator based on the maximally selected linear rank statistic was reasonable in terms of bias and standard deviation in the CR framework. The permutation-based test well satisfied type I error probability and provided higher power than the approximation-based test.

Acute cholecystitis - a cohort study in a real-world clinical setting (REWO study, NCT02796443).

BackgroundFor decades, the optimal timing of surgery for acute cholecystitis has been controversial. Recent meta-analyses and population-based studies favor early surgery. One recent large randomized trial has demonstrated that a delayed approach increases morbidity and cost compared to early surgery within 24 hours of hospital admission. Since cases of severe cholecystitis were excluded from this trial, we argue that these results do not reflect real-world clinical situations. From our point of view, these results were in contrast to the clinical experience with our patients; so, we decided to analyze critically all our patients with the null hypothesis that the patients treated with a delayed cholecystectomy after an acute cholecystitis have a similar or even better outcome than those treated with an early operative approach.Patients and methodsWe retrospectively analyzed clinical data from all patients with cholecystectomies in the period between January 2006 and September 2015. A total of 1,723 patients were categorized into four groups: early (n=138): urgent surgery of patients with acute cholecystitis within the first 72 hours of the onset of symptoms; intermediate (n=297): surgery of patients with acute cholecystitis within an average of 10 days after the onset of symptoms; delayed (n=427): initial non-surgical treatment of acute cholecystitis with surgery performed within 6–12 weeks of the onset of symptoms; and elective (n=868): cholecystectomy within a symptom-free interval of choice in patients with symptomatic cholecystolithiasis without signs of acute cholecystitis.ResultsIn a real-world scenario, early/intermediate cholecystectomy in acute cholecystitis was associated with a significant increase in morbidity and mortality (Clavien–Dindo score) compared to a delayed approach with surgery performed 6–12 weeks after the onset of symptoms. The adjusted linear rank statistics showed a decrease in the complication score with values of 2.29 in the early group, 0.48 in the intermediate group, –0.26 in the delayed group and –2.12 in the elective group. The results translate into a continuous decrease of the complication score from early over intermediate and delayed to the elective group.ConclusionThese results demonstrate that delayed cholecystectomy can be performed safely. In cases with severe cholecystitis, early and/or intermediate approaches still have a relatively high risk of morbidity and mortality.

On asymptotic normality of certain linear rank statistics

We consider asymptotic normality of linear rank statistics under various randomization rules met in clinical trials. Exposition relies on some general limit theorem given in McLeish (1974). Examples of applications include well known results as well as new ones. Though exposition is tightly attached to the context of clinical trials, the obtained results equally well apply to situations where randomization of similar kind takes place.

Weak convergence of the linear rank statistics under strong mixing conditions

We obtain the asymptotic distribution of the linear rank statistics under weak dependence. We consider a sequence of strong mixing random vectors with unequal dimensions and show the asymptotic normality of the rank statistics based on overall ranking.

Unbiased split variable selection for random survival forests using maximally selected rank statistics.

The most popular approach for analyzing survival data is the Cox regression model. The Cox model may, however, be misspecified, and its proportionality assumption may not always be fulfilled. An alternative approach for survival prediction is random forests for survival outcomes. The standard split criterion for random survival forests is the log-rank test statistic, which favors splitting variables with many possible split points. Conditional inference forests avoid this split variable selection bias. However, linear rank statistics are utilized by default in conditional inference forests to select the optimal splitting variable, which cannot detect non-linear effects in the independent variables. An alternative is to use maximally selected rank statistics for the split point selection. As in conditional inference forests, splitting variables are compared on the p-value scale. However, instead of the conditional Monte-Carlo approach used in conditional inference forests, p-value approximations are employed. We describe several p-value approximations and the implementation of the proposed random forest approach. A simulation study demonstrates that unbiased split variable selection is possible. However, there is a trade-off between unbiased split variable selection and runtime. In benchmark studies of prediction performance on simulated and real datasets, the new method performs better than random survival forests if informative dichotomous variables are combined with uninformative variables with more categories and better than conditional inference forests if non-linear covariate effects are included. In a runtime comparison, the method proves to be computationally faster than both alternatives, if a simple p-value approximation is used. Copyright © 2017 John Wiley & Sons, Ltd.

On the asymptotic normality of the R-estimators of the slope parameters of simple linear regression models with associated errors

ABSTRACTThe purpose of this paper is to prove, under mild conditions, the asymptotic normality of the rank estimator of the slope parameter of a simple linear regression model with stationary associated errors. This result follows from a uniform linearity property for linear rank statistics that we establish under general conditions on the dependence of the errors. We prove also a tightness criterion for weighted empirical process constructed from associated triangular arrays. This criterion is needed for the proofs which are based on that of Koul [Behavior of robust estimators in the regression model with dependent errors. Ann Stat. 1977;5(4):681–699] and of Louhichi [Louhichi S. Weak convergence for empirical processes of associated sequences. Ann Inst Henri Poincaré Probabilités Statist. 2000;36(5):547–567].

Rank Tests from Partially Ordered Data Using Importance and MCMC Sampling Methods

espanolExact tests, fuzzy p-values, Gibbs sampling, iterval censoring, linear extensions, linear rank statistics, perfect MCMC, proportional hazard model, topological sorting EnglishWe discuss distribution-free exact rank tests from partially ordered data that arise in various biological and other applications where the primary objective is to conduct testing of significance to assess the linear dependence or to compare different groups. The tests here are obtained by treating the usual rank statistics, based on the completely ordered data as “latent” or missing, and conceptualizing the “latent” p-value as the random probability under the null hypothesis of a test statistic that is as extreme, or more extreme, than the latent test statistics based on the completely ordered data. The latent p-value is then predicted by sampling linear extensions or the complete orderings that are consistent with the observed partially ordered data. The sampling methods explored here include importance sampling methods based on randomized topological sorting algorithms, Gibbs sampling methods, random-walk based Metropolis–Hasting sampling methods and random-walk based modern perfect Markov chain Monte Carlo sampling methods. We discuss running times of these sampling methods and their strength and weaknesses. A simulation experiment and three data examples are given. The simulation experiment illustrates how the exact rank tests from partially ordered data work when the desired result is known. The first data example concerns the light preference behavior of fruit flies and tests whether heterogeneity observed in average light-preference behavior can be explained by manipulations in serotonin signaling. The second one is a reanalysis of the lead absorption data in children of employees who worked in a lead battery factory and consolidates the results reported in Rosenbaum [Ann. Statist. 19 (1991) 1091–1097]. The third one reexamines the breast cosmesis data from Finkelstein [Biometrics 42 (1986) 845–854].

A moment generating function of a combination of linear rank tests and its asymptotic efficiency

When testing hypotheses in two-sample problems, the Lepage test has often been used to jointly test the location and scale parameters, and has been discussed by many authors over the years. The Lepage test was a combination of the Wilcoxon statistic and the Ansari–Bradley statistic. Various Lepage-type tests were proposed with discussions of an asymptotic relative efficiency (Duran et al., Biometrika 63:173–176, 1976; Goria, Stat Neerl 36:3–13, 1982), a robustness and a power comparison (Neuhauser, Commun Stat Theory Methods 29:67–78, 2000; Buning, J Appl Stat 29:907–924, 2002) and an adaptive test (Buning and Thadewald, J Stat Comput Sim 65:287–310, 2000). We derive an expression for the moment generating function of a linear combination of two linear rank statistics. As a suggested Lepage-type test, we use a combination of the generalized Wilcoxon statistic and the generalized Mood statistic. Deriving the exact critical value of the statistic can be difficult when the sample sizes are increasing. In this situation, an approximation method to the distribution function of the test statistic can be useful with a higher order moment. We use a moment-based approximation with an adjusted gamma polynomial to evaluate the upper tail probability of a Lepage-type test for a finite sample size. We determine the asymptotic efficiencies of the Lepage and Lepage-type tests for various distributions.

Rank tests in regression model based on minimum distance estimates

In this paper a new rank test in a linear regression model is introduced. The test statistic is based on a certain minimum distance estimator, however, unlike classical rank tests in regression it is not a simple linear rank statistic. Its exact distribution under the null hypothesis is derived, and further, the asymptotic distribution both under the null hypothesis and the local alternative is investigated. It is shown that the proposed test is applicable in measurement error models. Finally, a simulation study is conducted to show a good performance of the test. It has, in some situations, a greater power than the widely used Wilcoxon rank test.

Determining cutoff values of prognostic factors in survival data with competing risks

In clinical studies, patients are often classified into high or low risk groups based on prognostic factors related to survival outcomes. Using maximally selected linear rank statistics, several me...

Rank-based testing of equal survivorship based on cross-sectional survival data with or without prospective follow-up.

Existing linear rank statistics cannot be applied to cross-sectional survival data without follow-up since all subjects are essentially censored. However, partial survival information are available from backward recurrence times and are frequently collected from health surveys without prospective follow-up. Under length-biased sampling, a class of linear rank statistics is proposed based only on backward recurrence times without any prospective follow-up. When follow-up data are available, the proposed rank statistic and a conventional rank statistic that utilizes follow-up information from the same sample are shown to be asymptotically independent. We discuss four ways to combine these two statistics when follow-up is present. Simulations show that all combined statistics have substantially improved power compared with conventional rank statistics, and a Mantel-Haenszel test performed the best among the proposal statistics. The method is applied to a cross-sectional health survey without follow-up and a study of Alzheimer's disease with prospective follow-up.

Nonparametric Simultaneous Test Procedures

In this research we propose several nonparametric simultaneous test procedures for location and scale parameters. We construct test statistics based on linear rank statistics choosing a suitable combining function. We obtain the overall p -values by applying the permutation principle. We compare the efficiency amongst combining functions by obtaining empirical powers through a simulation study. We discuss some interesting aspects of our procedure as concluding remarks.