Exploiting Causal Structure for Robust Model Selection in Unsupervised Domain Adaptation

Abstract

In many real-world settings, such as healthcare, machine learning models are trained and validated on one labeled domain and tested or deployed on another, where feature distributions differ, i.e., there is <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">covariate shift</i> . When annotations are costly or prohibitive, an unsupervised domain adaptation (UDA) regime can be leveraged requiring only unlabeled samples in the target domain. Existing UDA methods are unable to factor in a model’s predictive loss based on predictions in the target domain and, therefore, suboptimally leverage density ratios of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">only the input covariates</i> in each domain. In this article, we propose a model selection method for leveraging model predictions on a target domain without labels by exploiting the domain invariance of causal structure. We assume or learn a causal graph from the source domain and select models that produce predicted distributions in the target domain that have the highest likelihood of fitting our causal graph. We thoroughly analyze our method under oracle knowledge using synthetic data. We then show on several real-world datasets, including several COVID-19 examples, that our method is able to improve on the state-of-the-art UDA algorithms for model selection.