Aspect coherence for graph-based semantic image labelling

Abstract

In image semantic segmentation a semantic category label is associated to each image pixel. This classification problem is characterised by pixel dependencies at different scales. On a small-scale pixel correlation is related to object instance sharing, whereas on a middle- and large scale to category co-presence and relative location constraints. The contribution of this study is two-fold. First, the authors present a framework that jointly learns category appearances and pixel dependencies at different scales. Small-scale dependencies are accounted by clustering pixels into larger patches via image oversegmentation. To tackle middle-scale dependencies a conditional random field (CRF) is built over the patches. A novel strategy to exploit local patch aspect coherence is used to impose an optimised structure in the graph to have exact and efficient inference. The second contribution is a method to account for full patch neighbourhoods without introducing loops in the graphical structures. ‘Weak neighbours’ are introduced, which are patches connected in the image but not in the inference graph. They are pre-classified according to their visual appearance and their category distribution probability is then used in the CRF inference step. Experimental evidence of the validity of the method shows improvements in comparison to other works in the field.