Covariate Shift Research Articles

Normalized-UNet Segmentation for COVID-19 Utilizing an Encoder-Decoder Connection Layer Block

The COVID-19 pandemic has had a huge influence on human lives all around the world. The virus spread quickly and impacted millions of individuals, resulting in a large number of hospitalizations and fatalities. The pandemic has also impacted economics, education, and social connections, among other aspects of life. Coronavirus-generated Computed Tomography (CT) scans have Regions of Interest (ROIs). The use of a modified U-Net model structure to categorize the region of interest at the pixel level is a promising strategy that may increase the accuracy of detecting COVID-19-associated anomalies in CT images. The suggested method seeks to detect and isolate ROIs in CT scans that show the existence of ground-glass opacity, which is frequent in COVID-19 patients. This can assist healthcare practitioners in identifying and monitoring illness development, as well as making treatment decisions. Scale U-Net is a strong U-Net design modification that can increase the performance of semantic segmentation tasks. Our model, Normalized-UNet, uses batch normalization after each convolutional layer to decrease the internal covariate shift, which dramatically improves the network's learning efficiency.

Re-Thinking the Effectiveness of Batch Normalization and Beyond.

Batch normalization (BN) is used by default in many modern deep neural networks due to its effectiveness in accelerating training convergence and boosting inference performance. Recent studies suggest that the effectiveness of BN is due to the Lipschitzness of the loss and gradient, rather than the reduction of internal covariate shift. However, questions remain about whether Lipschitzness is sufficient to explain the effectiveness of BN and whether there is room for vanilla BN to be further improved. To answer these questions, we first prove that when stochastic gradient descent (SGD) is applied to optimize a general non-convex problem, three effects will help convergence to be faster and better: (i) reduction of the gradient Lipschitz constant, (ii) reduction of the expectation of the square of the stochastic gradient, and (iii) reduction of the variance of the stochastic gradient. We demonstrate that vanilla BN only with ReLU can induce the three effects above, rather than Lipschitzness, but vanilla BN with other nonlinearities like Sigmoid, Tanh, and SELU will result in degraded convergence performance. To improve vanilla BN, we propose a new normalization approach, dubbed complete batch normalization (CBN), which changes the placement position of normalization and modifies the structure of vanilla BN based on the theory. It is proven that CBN can elicit all the three effects above, regardless of the nonlinear activation used. Extensive experiments on benchmark datasets CIFAR10, CIFAR100, and ILSVRC2012 validate that CBN makes the training convergence faster, and the training loss converges to a smaller local minimum than vanilla BN. Moreover, CBN helps networks with multiple nonlinear activations (Sigmoid, Tanh, ReLU, SELU, and Swish) achieve higher test accuracy steadily. Specifically, benefitting from CBN, the classification accuracies for networks with Sigmoid, Tanh, and SELU are boosted by more than 15.0%, 4.5%, and 4.0% on average, respectively, which is even comparable to the performance for ReLU.

Complex augmented representation network for transferable health prognosis of rolling bearing considering dynamic covariate shift

Complex augmented representation network for transferable health prognosis of rolling bearing considering dynamic covariate shift

Intelligent rolling bearing compound fault diagnosis based on frequency-domain Gramian angular field and convolutional neural networks with imbalanced data

The effective fault feature extraction is the core of rolling bearing fault diagnosis. However, rolling bearings usually operate in normal state and fault duration is very short, which will cause imbalance in fault diagnosis data, thus leading to difficulty in fault feature extraction and low diagnosis accuracy. Meanwhile, mutual interference between multiple fault responses will also lead to poor diagnosis performance. To solve these issues, a novel compound fault diagnosis method with imbalanced data based on frequency-domain Gramian angular field (FGAF) and convolutional neural networks optimized by instance normalization and efficient channel attention (IECNN) is proposed. Firstly, FGAF is adopted to map frequency-domain features of fault signals to the polar coordinate to obtain 2D FGAF feature spectrum. Secondly, an instance normalization module is established to reduce internal covariant shift caused by data distribution discrepancy and improve generalization ability. An efficient channel attention module is constructed to further excavate fault features and improve anti-interference ability. Finally, experiments are conducted under imbalanced dataset and imbalance intensified dataset, and the average accuracy of 99.91% and 99.92% were obtained, respectively, which shows the proposed method has better resistance to data imbalance.

On learning rates for regularized Nyström subsampling in unsupervised domain adaptation

The regularized Nyström subsampling is a popular approach for learning problems that deals with big data. We employ such technique in the context of the unsupervised domain adaptation problems with covariate shift assumption. Within the framework of the Reproducing Kernel Hilbert Space concept, an algorithm is constructed that is a combination of the Nyström subsampling and the two-steps iterated Tikhonov regularization. This approach allows significantly reduce the amount of computing resources involved and at the same time maintains the same learning rates as for the standard machine learning algorithms.

Evolving Domain Generalization Via Latent Structure-Aware Sequential Autoencoder.

Domain generalization (DG) refers to the problem of generalizing machine learning systems to out-of-distribution (OOD) data with knowledge learned from several provided source domains. Most prior works confine themselves to stationary and discrete environments to tackle such generalization issue arising from OOD data. However, in practice, many tasks in non-stationary environments (e.g.,autonomous-driving car system, sensor measurement) involve more complex and continuously evolving domain drift, emerging new challenges for model deployment. In this paper, we firstly formulate this setting as the problem of evolving domain generalization. To deal with the continuously changing domains, we propose MMD-LSAE, a novel framework that learns to capture the evolving patterns among domains for better generalization. Specifically, MMD-LSAE characterizes OOD data in non-stationary environments with two types of distribution shifts: covariate shift and concept shift, and employs deep autoencoder modules to infer their dynamics in latent space separately. In these modules, the inferred posterior distributions of latent codes are optimized to align with their corresponding prior distributions via minimizing maximum mean discrepancy (MMD). We theoretically verify that MMD-LSAE has the inherent capability to implicitly facilitate mutual information maximization, which can promote superior representation learning and improved generalization of the model. Furthermore, the experimental results on both synthetic and real-world datasets show that our proposed approach can consistently achieve favorable performance based on the evolving domain generalization setting.

Unsupervised Detection of Covariate Shift Due to Changes in EEG Headset Position: Towards an Effective Out-of-Lab Use of Passive Brain–Computer Interface

In the field of passive Brain–computer Interfaces (BCI), the need to develop systems that require rapid setup, suitable for use outside of laboratories is a fundamental challenge, especially now, that the market is flooded with novel EEG headsets with a good quality. However, the lack of control in operational conditions can compromise the performance of the machine learning model behind the BCI system. First, this study focuses on evaluating the performance loss of the BCI system, induced by a different positioning of the EEG headset (and of course sensors), so generating a variation in the control features used to calibrate the machine learning algorithm. This phenomenon is called covariate shift. Detecting covariate shift occurrences in advance allows for preventive measures, such as informing the user to adjust the position of the headset or applying specific corrections in new coming data. We used in this study an unsupervised Machine Learning model, the Isolation Forest, to detect covariate shift occurrence in new coming data. We tested the method on two different datasets, one in a controlled setting (9 participants), and the other in a more realistic setting (10 participants). In the controlled dataset, we simulated the movement of the EEG cap using different channel and reference configurations. For each test configuration, we selected a set of electrodes near the control electrodes. Regarding the realistic dataset, we aimed to simulate the use of the cap outside the laboratory, mimicking the removal and repositioning of the cap by a non-expert user. In both datasets, we recorded multiple test sessions for each configuration while executing a set of Workload tasks. The results obtained using the Isolation Forest model allowed the identification of covariate shift in the data, even with a 15-s recording sample. Moreover, the results showed a strong and significant negative correlation between the percentage of covariate shift detected by the method, and the accuracy of the passive BCI system (p-value < 0.01). This novel approach opens new perspectives for developing more robust and flexible BCI systems, with the potential to move these technologies towards out-of-the-lab use, without the need for supervision for use by a non-expert user.

Revisiting pretraining for semi-supervised learning in the low-label regime

Semi-supervised learning (SSL) addresses the lack of labeled data by exploiting large unlabeled data through pseudolabeling. However, in the extremely low-label regime, pseudo labels could be incorrect, a.k.a. the confirmation bias, and the pseudo labels will in turn harm the network training. Recent studies combined finetuning (FT) from pretrained weights with SSL to mitigate the challenges and claimed superior results in the low-label regime. In this work, we first show that the better pretrained weights brought in by FT account for the state-of-the-art performance, and importantly that they are universally helpful to off-the-shelf semi-supervised learners. We further argue that direct finetuning from pretrained weights is suboptimal due to covariate shift and propose a contrastive target pretraining step to adapt model weights towards target dataset. We carried out extensive experiments on both classification and segmentation tasks by doing target pretraining then followed by semi-supervised finetuning. The promising results validate the efficacy of target pretraining for SSL, in particular in the low-label regime.

Transfer learning methodology for machine learning based fault detection and diagnostics applied to building services

Machine Learning (ML) models for Fault Detection and Diagnosis (FDD) can automatically detect anomalies in the operation in large facilities or district heating networks and can help tackling energy wastes. Nevertheless, the development of ML-models is a costly and tedious task requiring large amounts of labelled data. Setting up ML-models for a high number of systems is effort and know-how intensive. However, assets like commercial buildings and district heating networks are constituted of systems with similar topologies. Transferring a ML model initially trained on a source system to a multitude of similar target systems, can help reducing the training costs and facilitating the scalability of ML-based FDD in those assets. To enable this, we have developed a methodology that assesses the potential for Transfer Learning (TL) from a source system to target systems by determining the covariate and concept shifts between the source and target domains and integrating the source model into the target system if the TL assessment is positive. We used a patented method for the model development, that combines two ML-models, that are initially trained on a source system by means of a feedback system. We implemented this methodology on district heating (DH) substations, as DH systems typically contain this kind of subsystems with similar topologies and have thus a high scalability potential for TL. Initial findings showed the effectiveness of TL in adapting the source model to the target domain, resulting in enhanced FDD capabilities with significantly reduced training efforts.

Stratified learning: A general‐purpose statistical method for improved learning under covariate shift

AbstractWe propose a simple, statistically principled, and theoretically justified method to improve supervised learning when the training set is not representative, a situation known as covariate shift. We build upon a well‐established methodology in causal inference and show that the effects of covariate shift can be reduced or eliminated by conditioning on propensity scores. In practice, this is achieved by fitting learners within strata constructed by partitioning the data based on the estimated propensity scores, leading to approximately balanced covariates and much‐improved target prediction. We refer to the overall method as Stratified Learning, or StratLearn. We demonstrate the effectiveness of this general‐purpose method on two contemporary research questions in cosmology, outperforming state‐of‐the‐art importance weighting methods. We obtain the best‐reported AUC (0.958) on the updated “Supernovae photometric classification challenge,” and we improve upon existing conditional density estimation of galaxy redshift from Sloan Digital Sky Survey (SDSS) data.

Consensus–relevance kNN and covariate shift mitigation

Consensus–relevance kNN and covariate shift mitigation

A Modified LeNet CNN for Breast Cancer Diagnosis in Ultrasound Images.

Convolutional neural networks (CNNs) have been extensively utilized in medical image processing to automatically extract meaningful features and classify various medical conditions, enabling faster and more accurate diagnoses. In this paper, LeNet, a classic CNN architecture, has been successfully applied to breast cancer data analysis. It demonstrates its ability to extract discriminative features and classify malignant and benign tumors with high accuracy, thereby supporting early detection and diagnosis of breast cancer. LeNet with corrected Rectified Linear Unit (ReLU), a modification of the traditional ReLU activation function, has been found to improve the performance of LeNet in breast cancer data analysis tasks via addressing the "dying ReLU" problem and enhancing the discriminative power of the extracted features. This has led to more accurate, reliable breast cancer detection and diagnosis and improved patient outcomes. Batch normalization improves the performance and training stability of small and shallow CNN architecture like LeNet. It helps to mitigate the effects of internal covariate shift, which refers to the change in the distribution of network activations during training. This classifier will lessen the overfitting problem and reduce the running time. The designed classifier is evaluated against the benchmarking deep learning models, proving that this has produced a higher recognition rate. The accuracy of the breast image recognition rate is 89.91%. This model will achieve better performance in segmentation, feature extraction, classification, and breast cancer tumor detection.

An information theoretic Bayesian uncertainty analysis of AEM systems over Menindee Lake, Australia

SUMMARY Long-range, active-source airborne electromagnetic (AEM) systems for near-surface conductivity imaging fall into two categories: helicopter (rotary-wing) borne or fixed-wing aircraft borne. A multitude of factors such as flying height, transmitter loop area and current, source waveforms, aerodynamic stability and data stacking times contribute to the geological resolvability of the subsurface. A comprehensive comparison of the relative merits of each system considering all such factors is difficult, but test flights over well-constrained subsurface geology with downhole induction logs are extremely useful for resolution studies. However, given the non-linear nature of the electromagnetic inverse problem, handling transmitter–receiver geometries in fixed-wing aircraft is especially challenging. As a consequence of this non-linearity, inspecting the closeness of downhole conductivities to deterministic inversion results is not sufficient for studying resolvability. A more comprehensive picture is provided by examining the variation in probability mass of the depth-wise Bayesian posterior conductivity distributions for each kind of AEM system within an information theoretic framework. For this purpose, probabilistic inversions of data must be carried out. Each acquiring system should fly over the same geology, survey noise levels must be measured and the same prior probabilities on conductivity must be used. With both synthetic models as well as real data from over the Menindee calibration range in New South Wales, Australia, we shed new light on the matter of AEM inverse model uncertainty. We do this using two information theoretic attributes derived from different Kullback–Leibler divergences—Bayesian information gain, and a strictly proper scoring rule, to assess posterior probabilities estimated by a novel Bayesian inversion scheme. The inversion marginalizes fixed-wing geometry attributes as generic nuisance parameters during Markov chain sampling. This is the first time-domain AEM study we know of, that compares nuisance marginalized subsurface posterior conductivities from a fixed-wing system, with rotary-wing derived posterior conductivities. We also compare field results with induction log data where available. Finally, we estimate the information gain in each case via a covariate shift adaptation technique that has not been used before in geophysical work. Our findings have useful implications in AEM system selection, as well as in the design of better deterministic AEM inversion algorithms.

Personalized Federated Learning for Institutional Prediction Model using Electronic Health Records: A Covariate Adjustment Approach.

Federated learning (FL) has attracted attention as a technology that allows multiple medical institutions to collaborate on AI without disclosing each other's patient data. However, FL has the challenge of being unable to robustly learn when the data of participating clients is non-independently and non-identically distributed (Non-IID). Personalized Federated Learning (PFL), which constructs a personalized model for each client, has been proposed as a solution to this problem. However, conventional PFL methods do not ensure the interpretability of personalization, specifically, the identification of which data samples are contributed to each personalized learning, which is important for AI in medical applications. In this study, we propose a novel PFL framework, Federated Adjustment of Covariate (FedCov), which acquires a propensity score model representing the covariate shift among clients through prior FL, then learns a final model by weighting the contribution of each training sample to PFL based on the estimated propensity score. This approach enables both the learning of personalized models through covariate adjustment and the visualization of the contribution of each client to PFL. FedCov was evaluated in the prediction of in-hospital mortality across 50 hospitals in the eICU Collaborative Research Database, achieving an ROC-AUC of 0.750. This result outperformed the AUCs in the 0.720-0.735 range achieved by conventional FL methods and was closest to the AUC of 0.754 achieved by centralized learning.Clinical Relevance- This study demonstrates the feasibility of providing sophisticated and personalized AI-driven clinical decision support to any medical institution through personalized federated learning.

Prediction sets adaptive to unknown covariate shift.

Predicting sets of outcomes-instead of unique outcomes-is a promising solution to uncertainty quantification in statistical learning. Despite a rich literature on constructing prediction sets with statistical guarantees, adapting to unknown covariate shift-a prevalent issue in practice-poses a serious unsolved challenge. In this article, we show that prediction sets with finite-sample coverage guarantee are uninformative and propose a novel flexible distribution-free method, PredSet-1Step, to efficiently construct prediction sets with an asymptotic coverage guarantee under unknown covariate shift. We formally show that our method is asymptotically probably approximately correct, having well-calibrated coverage error with high confidence for large samples. We illustrate that it achieves nominal coverage in a number of experiments and a data set concerning HIV risk prediction in a South African cohort study. Our theory hinges on a new bound for the convergence rate of the coverage of Wald confidence intervals based on general asymptotically linear estimators.

Covariate-Shift Generalization via Random Sample Weighting

Shifts in the marginal distribution of covariates from training to the test phase, named covariate-shifts, often lead to unstable prediction performance across agnostic testing data, especially under model misspecification. Recent literature on invariant learning attempts to learn an invariant predictor from heterogeneous environments. However, the performance of the learned predictor depends heavily on the availability and quality of provided environments. In this paper, we propose a simple and effective non-parametric method for generating heterogeneous environments via Random Sample Weighting (RSW). Given the training dataset from a single source environment, we randomly generate a set of covariate-determining sample weights and use each weighted training distribution to simulate an environment. We theoretically show that under appropriate conditions, such random sample weighting can produce sufficient heterogeneity to be exploited by common invariance constraints to find the invariant variables for stable prediction under covariate shifts. Extensive experiments on both simulated and real-world datasets clearly validate the effectiveness of our method.

Expert Data Augmentation in Imitation Learning (Student Abstract)

Behavioral Cloning (BC) is a simple and effective imitation learning algorithm, which suffers from compounding error due to covariate shift. One solution is to use enough data for training. However, the amount of expert demonstrations available is usually limited. So we propose an effective method to augment expert demonstrations to alleviate the problem of compounding error in BC. It operates by estimating the similarity of states and filtering out transitions that can go back to the states similar to ones in expert demonstrations during the process of sampling. The data filtered out along with original expert demonstrations are used for training. We evaluate the performance of our method on several Atari tasks and continuous MuJoCo control tasks. Empirically, BC trained with the augmented data significantly outperform BC trained with the original expert demonstrations.

Domain Generalised Faster R-CNN

Domain generalisation (i.e. out-of-distribution generalisation) is an open problem in machine learning, where the goal is to train a model via one or more source domains, that will generalise well to unknown target domains. While the topic is attracting increasing interest, it has not been studied in detail in the context of object detection. The established approaches all operate under the covariate shift assumption, where the conditional distributions are assumed to be approximately equal across source domains. This is the first paper to address domain generalisation in the context of object detection, with a rigorous mathematical analysis of domain shift, without the covariate shift assumption. We focus on improving the generalisation ability of object detection by proposing new regularisation terms to address the domain shift that arises due to both classification and bounding box regression. Also, we include an additional consistency regularisation term to align the local and global level predictions. The proposed approach is implemented as a Domain Generalised Faster R-CNN and evaluated using four object detection datasets which provide domain metadata (GWHD, Cityscapes, BDD100K, Sim10K) where it exhibits a consistent performance improvement over the baselines. All the codes for replicating the results in this paper can be found at https://github.com/karthikiitm87/domain-generalisation.git

SN-SAE: a new damage diagnosis method for CFRP using Lamb wave

The damage diagnosis of carbon fiber reinforced polymer (CFRP) using Lamb wave has been widely developed, but it is still a challenging task to obtain reliable damage diagnosis results by analysis of Lamb wave, the emergence of deep learning models provides an effective solution for this work. However, the internal covariate shift and overfitting exist in traditional deep networks. The SN-SAE (stochastic normalization-stacked autoencoder) deep neural network model is proposed by introducing stochastic normalization (SN) into stacked autoencoder (SAE). The signals of 28 different damage locations in the CFRP plate provided by the open platform were processed by SN-SAE, and the damage diagnosis at different locations was achieved. The validity of SN-SAE was further verified by data obtained through building an experimental platform. The results demonstrated that the SN-SAE model can achieve high test accuracy with only 15% of the data samples as training with limited data sample, which provides a simple and effective solution for damage diagnosis of composite plates.

Posterior Covariance Information Criterion for Weighted Inference.

For predictive evaluation based on quasi-posterior distributions, we develop a new information criterion, the posterior covariance information criterion (PCIC). PCIC generalizes the widely applicable information criterion (WAIC) so as to effectively handle predictive scenarios where likelihoods for the estimation and the evaluation of the model may be different. A typical example of such scenarios is the weighted likelihood inference, including prediction under covariate shift and counterfactual prediction. The proposed criterion uses a posterior covariance form and is computed by using only one Markov chain Monte Carlo run. Through numerical examples, we demonstrate how PCIC can apply in practice. Further, we show that PCIC is asymptotically unbiased to the quasi-Bayesian generalization error under mild conditions in weighted inference with both regular and singular statistical models.