Uncertainty-Aware Scene Graph Generation

Abstract

As a challenging task in computer vision, Scene graph generation (SGG) aims to model the underlying semantic relationships among objects in a given image for scene understanding. Due to the increasing scale and subjectivity, the annotations of existing SGG benchmarks inevitably suffer from some uncertainty issues, resulting in the models hardly learning the relationships comprehensively. In this work, we address the uncertainty from the perspectives of both classifier parameters and relationship labels. On one hand, we handle the classifier uncertainty via learning a Bayesian classifier reparameterization, of which the weights are sampled from a latent space spanned with a prior distribution. On the other hand, we assume that each relationship label is sampled from a latent label space and mitigate the label uncertainty via estimating the latent relationship distribution. As a result, the distribution of the classifier parameters are comprehensively learned under the supervision of the estimated relationship labels, thus improving the model’s generalization ability. Experimental results on the popular benchmark demonstrate that the proposed strategies significantly improve different baseline models on different SGG tasks.