Integrative conformal p-values for out-of-distribution testing with labelled outliers

Abstract

Abstract This paper presents a conformal inference method for out-of-distribution testing that leverages side information from labelled outliers, which are commonly underutilized or even discarded by conventional conformal p-values. This solution is practical and blends inductive and transductive inference strategies to adaptively weight conformal p-values, while also automatically leveraging the most powerful model from a collection of one-class and binary classifiers. Further, this approach leads to rigorous false discovery rate control in multiple testing when combined with a conditional calibration strategy. Extensive numerical simulations show that the proposed method outperforms existing approaches.