Visual attention pooling and understanding the structural similarity index in multi-scale analysis

Abstract

We present a novel spatial pooling strategy and the results of an extensive multi-scale analysis of the well-known structural similarity index metric (SSIM) for objective image quality evaluation. We show, in contrast with some previous studies, that even relatively simple perceptual importance pooling strategies can significantly improve objective metric performance evaluated as the correlation with subjective quality assessment. In particular, we define an attention and quality driven pooling mechanism that focuses structural comparisons within the SSIM model to only those pixels exhibiting significant structural degradations. We show that optimal objective metric performance is achieved over very sparse spatial domains indeed that ignore most of the signal data. We also investigate an explicit breakdown of the structural models within SSIM and show that in combination with the proposed attention and quality driven pooling some of these models represent well performing metrics in their own right, when applied at appropriate scale for which there may not be a single optimal value. Our experiments demonstrate that the augmented SSIM metric using the proposed pooling model provides performance advantage on an extensive LIVE dataset covering hundreds of degraded images and 5 different distortion types compared to both conventional SSIM and state-of-the-art objective quality metrics.