Mitigating bias in facial analysis systems by incorporating label diversity

Abstract

Facial analysis models are increasingly applied in real-world applications that significantly impact people’s lives. However, as literature has shown, models that automatically classify facial attributes might exhibit algorithmic discrimination behavior concerning protected groups, potentially negatively impacting individuals and society. It is, therefore, critical to develop techniques that can mitigate unintended biases in facial classifiers. Hence, this work introduces a novel learning method combining subjective human-based labels and objective annotations based on mathematical definitions of facial traits. Specifically, we generate new objective annotations from two large-scale human-annotated datasets, each capturing a different perspective of the analyzed facial trait. We then propose an ensemble learning method, which combines individual models trained on different types of annotations. We provide an in-depth analysis of the annotation procedure as well as the datasets distribution. We next show how much each individual model, each trained on a different set of annotations, contributes to the final ensemble. Finally, we empirically show that, by incorporating label diversity in the model pipeline, our method successfully mitigates unintended biases while maintaining significant accuracy on the downstream tasks. Our method is generic and can be applied to different types of facial tasks.