U-Shaped Transformer-Based 360-Degree No Reference Image Quality Assessment

Abstract

Thanks to creative rendering and display techniques, 360-degree images can provide a more immersive and interactive experience for streaming users. However, such features make the perceptual characteristics of 360-degree images more complex than those of fixed-view images, making it impossible to achieve a comprehensive and accurate image quality assessment (IQA) task using a simple stack of pre-processing, post-processing, compression, and rendering tasks. In order to thoroughly learn global and local features in 360-degree images, reduce the complexity of multichannel neural network models and simplify the training process, this paper proposes a user-aware joint architecture and an efficient converter dedicated to 360-degree no-reference (NR) IQA. The input of the proposed method is a 360-degree cubic mapping projection (CMP) image. In addition, the proposed 360-degree NRIQA method includes a non-overlapping self-attentive selection module based on a dominant map and a feature extraction module based on a U-shaped transformer (U-former) to address perceptual region significance and projection distortion. The transformer-based architecture and the weighted averaging technique are jointly used to predict local perceptual quality. Experimental results obtained on widely used databases show that the proposed model outperforms other state-of-the-art methods in the case of NR 360-degree image quality assessment. In addition, cross-database evaluation and ablation studies demonstrate the intrinsic robustness and generalization of the proposed model.