Dataset Shift Research Articles

Kernel-Based Domain-Invariant Feature Selection in Hyperspectral Images for Transfer Learning

This paper presents a kernel-based feature selection method for the classification of hyperspectral images. The proposed method aims at selecting a subset of the original features that are both 1) relevant (discriminant) for the considered classification problem, i.e., preserve the functional relationship between input and output variables, and 2) invariant (stable) across different domains, i.e., minimize the data-set shift between the source and the target domains. Domains can be associated with hyperspectral images collected either on different geographical areas or on the same area at different times. We propose a novel measure of data-set shift for evaluating the domain stability, which computes the distance of the conditional distributions between the source and target domains in a reproducing kernel Hilbert space. Such a measure is defined on the basis of the kernel embeddings of the conditional distributions resulting in a nonparametric approach that does not require estimating the distribution of the classes. The adopted search strategy is based on a multiobjective optimization algorithm, which optimizes the two terms of the criterion function for the estimation of the Pareto-optimal solutions. This results in an effective approach of performing feature selection in a transfer learning setting. The experimental results obtained on two hyperspectral images show the effectiveness of the proposed method in selecting features with high generalization capabilities.

Fieller Stability Measure: a novel model-dependent backtesting approach

Dataset shift is present in almost all real-world applications, since most of them are constantly dealing with changing environments. Detecting fractures in datasets on time allows recalibrating the models before a significant decrease in the model’s performance is observed. Since small changes are normal in most applications and do not justify the efforts that a model recalibration requires, we are only interested in identifying those changes that are critical for the correct functioning of the model. In this work we propose a model-dependent backtesting strategy designed to identify significant changes in the covariates, relating a confidence zone of the change to a maximal deviance measure obtained from the coefficients of the model. Using logistic regression as a predictive approach, we performed experiments on simulated data, and on a real-world credit scoring dataset. The results show that the proposed method has better performance than traditional approaches, consistently identifying major changes in variables while taking into account important characteristics of the problem, such as sample sizes and variances, and uncertainty in the coefficients.

DDoS attack protection in the era of cloud computing and Software-Defined Networking

Cloud computing has become the real trend of enterprise IT service model that offers cost-effective and scalable processing. Meanwhile, Software-Defined Networking (SDN) is gaining popularity in enterprise networks for flexibility in network management service and reduced operational cost. There seems a trend for the two technologies to go hand-in-hand in providing an enterprise’s IT services. However, the new challenges brought by the marriage of cloud computing and SDN, particularly the implications on enterprise network security, have not been well understood. This paper sets to address this important problem.We start by examining the security impact, in particular, the impact on DDoS attack defense mechanisms, in an enterprise network where both technologies are adopted. We find that SDN technology can actually help enterprises to defend against DDoS attacks if the defense architecture is designed properly. To that end, we propose a DDoS attack mitigation architecture that integrates a highly programmable network monitoring to enable attack detection and a flexible control structure to allow fast and specific attack reaction. To cope with the new architecture, we propose a graphic model based attack detection system that can deal with the dataset shift problem. The simulation results show that our architecture can effectively and efficiently address the security challenges brought by the new network paradigm and our attack detection system can effectively report various attacks using real-world network traffic.

EWMA model based shift-detection methods for detecting covariate shifts in non-stationary environments

Dataset shift is a very common issue wherein the input data distribution shifts over time in non-stationary environments. A broad range of real-world systems face the challenge of dataset shift. In such systems, continuous monitoring of the process behavior and tracking the state of shift are required in order to decide about initiating adaptive corrections in a timely manner. This paper presents novel methods for covariate shift-detection tests based on a two-stage structure for both univariate and multivariate time-series. The first stage works in an online mode and it uses an exponentially weighted moving average (EWMA) model based control chart to detect the covariate shift-point in non-stationary time-series. The second stage validates the shift-detected by first stage using the Kolmogorov–Smirnov statistical hypothesis test (K–S test) in the case of univariate time-series and the Hotelling T-Squared multivariate statistical hypothesis test in the case of multivariate time-series. Additionally, several orthogonal transformations and blind source separation algorithms are investigated to counteract the adverse effect of cross-correlation in multivariate time-series on shift-detection performance. The proposed methods are suitable to be run in real-time. Their performance is evaluated through experiments using several synthetic and real-world datasets. Results show that all the covariate shifts are detected with much reduced false-alarms compared to other methods.

A review of microarray datasets and applied feature selection methods

Microarray data classification is a difficult challenge for machine learning researchers due to its high number of features and the small sample sizes. Feature selection has been soon considered a de facto standard in this field since its introduction, and a huge number of feature selection methods were utilized trying to reduce the input dimensionality while improving the classification performance. This paper is devoted to reviewing the most up-to-date feature selection methods developed in this field and the microarray databases most frequently used in the literature. We also make the interested reader aware of the problematic of data characteristics in this domain, such as the imbalance of the data, their complexity, or the so-called dataset shift. Finally, an experimental evaluation on the most representative datasets using well-known feature selection methods is presented, bearing in mind that the aim is not to provide the best feature selection method, but to facilitate their comparative study by the research community.

Virtual and Real World Adaptation for Pedestrian Detection.

Pedestrian detection is of paramount interest for many applications. Most promising detectors rely on discriminatively learnt classifiers, i.e., trained with annotated samples. However, the annotation step is a human intensive and subjective task worth to be minimized. By using virtual worlds we can automatically obtain precise and rich annotations. Thus, we face the question: can a pedestrian appearance model learnt in realistic virtual worlds work successfully for pedestrian detection in real-world images? Conducted experiments show that virtual-world based training can provide excellent testing accuracy in real world, but it can also suffer the data set shift problem as real-world based training does. Accordingly, we have designed a domain adaptation framework, V-AYLA, in which we have tested different techniques to collect a few pedestrian samples from the target domain (real world) and combine them with the many examples of the source domain (virtual world) in order to train a domain adapted pedestrian classifier that will operate in the target domain. V-AYLA reports the same detection accuracy than when training with many human-provided pedestrian annotations and testing with real-world images of the same domain. To the best of our knowledge, this is the first work demonstrating adaptation of virtual and real worlds for developing an object detector.

FAULT DETECTION FOR THE CLASS IMBALANCE PROBLEM IN SEMICONDUCTOR MANUFACTURING PROCESSES

In the semiconductor manufacturing process, fault detection which aims at constructing a decision tool to maintain high process yields is a major step of the process control. Unfortunately, the class imbalance in the modern semiconductor industry makes feature selection for fault detection quite challenging. However, the characteristic has usually been ignored in the open literatures. This paper analyzes the challenge and indicates some of the reasons are due to the dataset shift, the small samples and the class overlapping caused by the class imbalance. To cope with the problems, a new feature selection approach is proposed, which combines the global and local resampling and named ensemble manifold sensitive margin fisher analysis (EMSMFA). Our approach consists of three key components: (1) At the global level, the bagging-based ensemble model is used to overcome the overfitting caused by the data shift; (2) At the local level, the manifold-based oversampling named the weighted synthetic minority oversampling technique (WSMOTE) is proposed to solve the small samples problem in the minority class; (3) The sensitive margin fisher analysis (SMFA) is used to solve the challenge caused by the class overlapping. The proposed fault detection method is demonstrated through its application to the semiconductor wafer fabrication process. The experimental results confirm the EMSMFA improves the performance of fault detection.

On the importance of the validation technique for classification with imbalanced datasets: Addressing covariate shift when data is skewed

In the field of Data Mining, the estimation of the quality of the learned models is a key step in order to select the most appropriate tool for the problem to be solved. Traditionally, a k-fold validation technique has been carried out so that there is a certain degree of independency among the results for the different partitions. In this way, the highest average performance will be obtained by the most robust approach. However, applying a “random” division of the instances over the folds may result in a problem known as dataset shift, which consists in having a different data distribution between the training and test folds.In classification with imbalanced datasets, in which the number of instances of one class is much lower than the other class, this problem is more severe. The misclassification of minority class instances due to an incorrect learning of the real boundaries caused by a not well fitted data distribution, truly affects the measures of performance in this scenario. Regarding this fact, we propose the use of a specific validation technique for the partitioning of the data, known as “Distribution optimally balanced stratified cross-validation” to avoid this harmful situation in the presence of imbalance. This methodology makes the decision of placing close-by samples on different folds, so that each partition will end up with enough representatives of every region.We have selected a wide number of imbalanced datasets from KEEL dataset repository for our study, using several learning techniques from different paradigms, thus making the conclusions extracted to be independent of the underlying classifier. The analysis of the results has been carried out by means of the proper statistical study, which shows the goodness of this approach for dealing with imbalanced data.

An insight into classification with imbalanced data: Empirical results and current trends on using data intrinsic characteristics

Training classifiers with datasets which suffer of imbalanced class distributions is an important problem in data mining. This issue occurs when the number of examples representing the class of interest is much lower than the ones of the other classes. Its presence in many real-world applications has brought along a growth of attention from researchers.We shortly review the many issues in machine learning and applications of this problem, by introducing the characteristics of the imbalanced dataset scenario in classification, presenting the specific metrics for evaluating performance in class imbalanced learning and enumerating the proposed solutions. In particular, we will describe preprocessing, cost-sensitive learning and ensemble techniques, carrying out an experimental study to contrast these approaches in an intra and inter-family comparison.We will carry out a thorough discussion on the main issues related to using data intrinsic characteristics in this classification problem. This will help to improve the current models with respect to: the presence of small disjuncts, the lack of density in the training data, the overlapping between classes, the identification of noisy data, the significance of the borderline instances, and the dataset shift between the training and the test distributions. Finally, we introduce several approaches and recommendations to address these problems in conjunction with imbalanced data, and we will show some experimental examples on the behavior of the learning algorithms on data with such intrinsic characteristics.

Multitask Remote Sensing Data Classification

Many remote sensing data processing problems are inherently constituted by several tasks that can be solved either individually or jointly. For instance, each image in a multitemporal classification setting could be taken as an individual task. Here, the relation to previous acquisitions should be properly considered because of the nonstationary behavior of temporal, spatial, and angular image features which gives rise to distribution changes. This phenomenon is known as covariate shift. Additionally, when labeled data are scarce or expensive to obtain, the small sample-set problem arises, which makes solving the problems independently in each domain difficult. Multitask learning (MTL) aims at jointly solving a set of prediction problems by sharing information across tasks. This paper introduces MTL in remote sensing data classification. The proposed methods alleviate the data set shift by imposing cross-information in the classifiers through matrix regularization. We consider the support vector machine (SVM) as the core learner and two different regularization schemes: 1) the inclusion of relational operators between tasks and 2) the pairwise Euclidean distance of the predictors in the Hilbert space. These methods rely on simple and intuitive modifications of the kernel used in the standard SVM. Experiments are conducted in three challenging remote sensing problems: cloud screening from multispectral images, land-mine detection using radar data, and multitemporal and multisource image classification. The pairwise method consistently outperforms standard independent and aggregate approaches by about +2% to 4% in all problems at no additional cost. Also, the solutions found give us information about the distribution shift among tasks.

Study on the impact of partition-induced dataset shift on k-fold cross-validation.

Cross-validation is a very commonly employed technique used to evaluate classifier performance. However, it can potentially introduce dataset shift, a harmful factor that is often not taken into account and can result in inaccurate performance estimation. This paper analyzes the prevalence and impact of partition-induced covariate shift on different k-fold cross-validation schemes. From the experimental results obtained, we conclude that the degree of partition-induced covariate shift depends on the cross-validation scheme considered. In this way, worse schemes may harm the correctness of a single-classifier performance estimation and also increase the needed number of repetitions of cross-validation to reach a stable performance estimation.

Analysis of preprocessing vs. cost-sensitive learning for imbalanced classification. Open problems on intrinsic data characteristics

Class imbalance is among the most persistent complications which may confront the traditional supervised learning task in real-world applications. The problem occurs, in the binary case, when the number of instances in one class significantly outnumbers the number of instances in the other class. This situation is a handicap when trying to identify the minority class, as the learning algorithms are not usually adapted to such characteristics.The approaches to deal with the problem of imbalanced datasets fall into two major categories: data sampling and algorithmic modification. Cost-sensitive learning solutions incorporating both the data and algorithm level approaches assume higher misclassification costs with samples in the minority class and seek to minimize high cost errors. Nevertheless, there is not a full exhaustive comparison between those models which can help us to determine the most appropriate one under different scenarios.The main objective of this work is to analyze the performance of data level proposals against algorithm level proposals focusing in cost-sensitive models and versus a hybrid procedure that combines those two approaches. We will show, by means of a statistical comparative analysis, that we cannot highlight an unique approach among the rest. This will lead to a discussion about the data intrinsic characteristics of the imbalanced classification problem which will help to follow new paths that can lead to the improvement of current models mainly focusing on class overlap and dataset shift in imbalanced classification.

On the dataset shift problem in software engineering prediction models

A core assumption of any prediction model is that test data distribution does not differ from training data distribution. Prediction models used in software engineering are no exception. In reality, this assumption can be violated in many ways resulting in inconsistent and non-transferrable observations across different cases. The goal of this paper is to explain the phenomena of conclusion instability through the dataset shift concept from software effort and fault prediction perspective. Different types of dataset shift are explained with examples from software engineering, and techniques for addressing associated problems are discussed. While dataset shifts in the form of sample selection bias and imbalanced data are well-known in software engineering research, understanding other types is relevant for possible interpretations of the non-transferable results across different sites and studies. Software engineering community should be aware of and account for the dataset shift related issues when evaluating the validity of research outcomes.

A unifying view on dataset shift in classification

The field of dataset shift has received a growing amount of interest in the last few years. The fact that most real-world applications have to cope with some form of shift makes its study highly relevant. The literature on the topic is mostly scattered, and different authors use different names to refer to the same concepts, or use the same name for different concepts. With this work, we attempt to present a unifying framework through the review and comparison of some of the most important works in the literature. ► Presentation of a unifying framework for the field of dataset shift, focusing on classification. ► Analysis of the terminology used in the most relevant works of the field. ► Formal definitions for each of the concepts appearing in the study of dataset shift, including sample selection bias.

Electrocardiographic abnormalities and autonomic dysfunction in Guillain-Barre syndrome

<b>2724</b> <h3><b>Introduction:</b></h3> In developing artificial intelligence (AI) algorithms for nuclear medicine applications, several pitfalls are frequently encountered which can impede progress, lead to erroneous findings, and ultimately limit the clinical utility of algorithms. The AI Task Force of the Society of Nuclear Medicine and Molecular Imaging has identified pitfalls that commonly afflict AI algorithm development and we provide suggestions on how to best avoid them. <h3><b>Methods:</b></h3> Here we address three of the most common and detrimental pitfalls that affect AI algorithm development, including for applications within nuclear medicine: 1) exaggerated estimates of algorithm performance (reproducibility); 2) algorithms with acceptable performance in only limited populations (generalizability); and 3) algorithms that are poorly matched to the clinical need (suitability). <h3><b>Results:</b></h3> To address the challenge of poor reproducibility (i.e., the inability to replicate previous research findings), algorithms must be evaluated in datasets that are independent from the training data. Developmental datasets should be partitioned into training and holdout testing cohorts and performance measurements should be reported in the withheld test cohort. For all but large datasets, cross-validation methods such as nested cross validation should be used. Data leakage, in which information from the test set influences the model’s training, should be avoided. For hypothesis testing, statistical power analysis should be used to determine the sample size of the test cohort, and preplanned statistical analysis can help avoid “p-hacking”. Codes and models should be made publicly available and be sufficient to enable replication. The reporting of results in literature should be thorough and transparent, and we recommend the use of reporting checklists. To address the challenge of poor generalizability, developmental datasets should be collected from diverse sources, representing the anticipated variability of the real world clinical population, including images from different scanner technologies. Data samples should be collected from groups that might be vulnerable to biases and subgroup analyses should be performed. Dataset shift should be evaluated, in which the trained model is evaluated on external cohorts from different institutions. To address the challenge of poor suitability, development teams should include not just AI experts but also clinical domain experts and stakeholders, including physicians and technologists, so that the algorithm’s output can be best aligned with the clinical need. For applications in which algorithm explainability is deemed beneficial to users, algorithms should be designed so that the model’s predictions are interpretable and explainable. The confidence associated with the algorithm output should be provided whenever possible. <h3><b>Conclusions:</b></h3> The recent growth of interest in AI has led to many promising technologies in nuclear medicine but also poses several challenges, including algorithms with poor reproducibility, poor generalizability, and poor suitability for clinical tasks. By following best practices for AI algorithm development, these challenges can be overcome and developers can realize the promise of AI while avoiding the pitfalls.