Cubemap Projection Research Articles

Multi-Projection Fusion and Refinement Network for Salient Object Detection in 360° Omnidirectional Image.

Salient object detection (SOD) aims to determine the most visually attractive objects in an image. With the development of virtual reality (VR) technology, 360° omnidirectional image has been widely used, but the SOD task in 360° omnidirectional image is seldom studied due to its severe distortions and complex scenes. In this article, we propose a multi-projection fusion and refinement network (MPFR-Net) to detect the salient objects in 360° omnidirectional image. Different from the existing methods, the equirectangular projection (EP) image and four corresponding cube-unfolding (CU) images are embedded into the network simultaneously as inputs, where the CU images not only provide supplementary information for EP image but also ensure the object integrity of cube-map projection. In order to make full use of these two projection modes, a dynamic weighting fusion (DWF) module is designed to adaptively integrate the features of different projections in a complementary and dynamic manner from the perspective of inter and intrafeatures. Furthermore, in order to fully explore the way of interaction between encoder and decoder features, a filtration and refinement (FR) module is designed to suppress the redundant information of the feature itself and between the features. Experimental results on two omnidirectional datasets demonstrate that the proposed approach outperforms the state-of-the-art methods both qualitatively and quantitatively. The code and results can be found from the link of https://rmcong.github.io/proj_MPFRNet.html.

AFNet: Asymmetric fusion network for monocular panorama depth estimation

Panoramic depth estimation is a crucial technique in 360°perception, which has gained great popularity in recent years due to its ability to provide complete visual coverage of the scene. Panoramic images are commonly presented in the form of equirectangular projection. However, this projection exhibits significant distortions towards the poles, making it unsuitable to directly apply traditional depth estimation frameworks for perspective imaging. The cubemap projection offers distortion-free images on each face but exhibits obvious discontinuities between adjacent faces. Therefore, according to the respective advantages of these two projection methods, we propose an asymmetric fusion network (AFNet), where the fusion information is only fed back to the equirectangular projection branch during the encoding stage to guide the subsequent encoding. We also design a dual projection feature fusion (DPFF) module in our framework to better combine the strengths of both projections. Furthermore, we introduce an edge information enhancement (EIE) module to enhance the edge contours of the final depth map. Experiments on four representative datasets (including both synthetic and real-world scenarios) show that our proposed AFNet achieves favorable performance compared with existing methods.

Omnidirectional Image Super-resolution via Bi-projection Fusion

With the rapid development of virtual reality, omnidirectional images (ODIs) have attracted much attention from both the industrial community and academia. However, due to storage and transmission limitations, the resolution of current ODIs is often insufficient to provide an immersive virtual reality experience. Previous approaches address this issue using conventional 2D super-resolution techniques on equirectangular projection without exploiting the unique geometric properties of ODIs. In particular, the equirectangular projection (ERP) provides a complete field-of-view but introduces significant distortion, while the cubemap projection (CMP) can reduce distortion yet has a limited field-of-view. In this paper, we present a novel Bi-Projection Omnidirectional Image Super-Resolution (BPOSR) network to take advantage of the geometric properties of the above two projections. Then, we design two tailored attention methods for these projections: Horizontal Striped Transformer Block (HSTB) for ERP and Perspective Shift Transformer Block (PSTB) for CMP. Furthermore, we propose a fusion module to make these projections complement each other. Extensive experiments demonstrate that BPOSR achieves state-of-the-art performance on omnidirectional image super-resolution. The code is available at https://github.com/W-JG/BPOSR.

Local-to-Global Panorama Inpainting for Locale-Aware Indoor Lighting Prediction.

Predicting panoramic indoor lighting from a single perspective image is a fundamental but highly ill-posed problem in computer vision and graphics. To achieve locale-aware and robust prediction, this problem can be decomposed into three sub-tasks: depth-based image warping, panorama inpainting and high-dynamic-range (HDR) reconstruction, among which the success of panorama inpainting plays a key role. Recent methods mostly rely on convolutional neural networks (CNNs) to fill the missing contents in the warped panorama. However, they usually achieve suboptimal performance since the missing contents occupy a very large portion in the panoramic space while CNNs are plagued by limited receptive fields. The spatially-varying distortion in the spherical signals further increases the difficulty for conventional CNNs. To address these issues, we propose a local-to-global strategy for large-scale panorama inpainting. In our method, a depth-guided local inpainting is first applied on the warped panorama to fill small but dense holes. Then, a transformer-based network, dubbed PanoTransformer, is designed to hallucinate reasonable global structures in the large holes. To avoid distortion, we further employ cubemap projection in our design of PanoTransformer. The high-quality panorama recovered at any locale helps us to capture spatially-varying indoor illumination with physically-plausible global structures and fine details.

Visual perception-oriented quality assessment for high dynamic range stereoscopic omnidirectional video system

High dynamic range stereoscopic omnidirectional video (HSOV) system provides users with an immersive visual experience, but the distortions generated in the process of visual content generation, coding, and visualization also brings challenges to visual quality assessment. This paper proposes a visual perception-oriented quality metric for HSOV system. Considering the computational efficiency, for frames within each second, several key frames are selected so that features reflecting distortions can be extracted from the viewports and equi-angular cubemap (EAC) projection surfaces of these frames. In viewport feature extraction module, the sum and difference maps of the left and right views of the key frames are used to simulate the binocular perception. Considering that distortions will affect the structure, color and statistical properties of the image, the gradient, local binary pattern and statistical features are extracted, respectively. To reduce projection distortion, EAC projection is used to instead of other projections normally used in quality assessment, and the Sigma domain feature extraction and color feature extraction are performed on EAC projection map. In temporal feature extraction module, wavelet decomposition is used to perform temporal domain bandpass filtering on partial continuous frames to calculate the naturalness features. Finally, the above features of the frames at different scales are pooled through random forest model to predicate the quality score of HSOV. Experimental results on the NBU-HSOVD dataset show that the proposed metric is superior to some blind image/video quality assessment metrics and has higher consistency with the human visual perception.

Perception-Oriented U-Shaped Transformer Network for 360-Degree No-Reference Image Quality Assessment

Generally, 360-degree images have absolute senses of reality and three-dimensionality, providing a wide range of immersive interactions. Due to the novel rendering and display technology of 360-degree images, they have more complex perceptual characteristics than other images. It is challenging to perform comprehensive image quality assessment (IQA) learning by simply stacking multichannel neural network architectures for pre/postprocessing, compression, and rendering tasks. To thoroughly learn the global and local features in 360-degree images, reduce the complexity of multichannel neural network models and simplify the training process, this paper proposes a joint architecture with user perception and an efficient transformer dedicated to 360-degree no-reference (NR) IQA. The input of the proposed method is a 360-degree cube map projection (CMP) image. Furthermore, the proposed 360-degree NRIQA method includes a saliency map-based non-overlapping self-attention selection module and a U-shaped transformer (U-former)-based feature extraction module to account for perceptual region importance and projection distortion. The transformer-based architecture and the weighted average technique are jointly utilized for predicting local perceptual quality. Experimental results obtained on widely used databases show that the proposed model outperforms other state-of-the-art methods in NR 360-degree image quality evaluation cases. Furthermore, a cross-database evaluation and an ablation study also demonstrate the inherent robustness and generalization ability of the proposed model.

Texture-Aware Spherical Rotation for High Efficiency Omnidirectional Intra Video Coding

To adapt to the existing video coding standards, omnidirectional videos are usually projected from Three-Dimensional (3D) sphere to Two-Dimensional (2D) plane. However, this projection will cause geometrical stretching distortion and boundary discontinuity, which may degrade coding efficiency. In this paper, we present a Spherical Rotation based Omnidirectional Video Coding (SROVC) method, which exploits the textural properties of omnidirectional videos with spherical rotation. Firstly, SROVC framework is presented and Full-traversal Spherical Rotation (FSR) is developed to derive the optimal rotation angle with frame-level Rate Distortion Optimization (RDO). Secondly, to achieve comparable coding gains and lower computational complexity when compared with FSR, a Texture-aware Spherical Rotation (TSR) method is proposed to predict the rotation angle. Finally, to further reduce complexity and maintain coding efficiency, a Group-oriented TSR (G-TSR) approach is presented, in which the group length is statistically determined. Extensive experiments demonstrate that the proposed TSR and G-TSR schemes can achieve bit rate reductions up to 4.38%, 0.94% and 0.89% on average for CubeMap Projection (CMP) based high efficiency omnidirectional video coding. Additionally, the TSR scheme achieves bit rate saving from 0.91% to 1.19% on average under three more CMP-based projection formats, and 1.89% for joint rotation of X, Y, and Z axes.

Multi-Angle Projection Based Blind Omnidirectional Image Quality Assessment

Most of the existing blind omnidirectional image quality assessment (BOIQA) methods are based on data-driven approach where the end-to-end neural network or deep learning tools are mainly used for feature extraction. However, it usually lacks interpretability and is difficult to discover the perceptual mechanism behind. In this paper, from the perspective of perception modeling, we propose a novel multi-angle projection based BOIQA (MP-BOIQA) method. Considering the omnibearing and near eye display characteristics with head mounted display, multiple color cubemap projection images with respect to different viewpoints are grouped as the color omnidirectional distortion (COD) units so as to simulate the user’s viewing behavior in subjective quality assessment. In the designed multi-angle projection based feature extractor, tensor decomposition is implemented on each COD unit for dimensionality reduction, and piecewise exponential fitting is used to get the distribution of mean subtracted contrast normalized coefficients of the unit’s feature matrices in tensor domain. Finally, the extracted features are pooled with random forest. The experimental results on three omnidirectional image quality datasets show that the MP-BOIQA method can deliver highly competitive performance compared with some representative full-reference quality assessment methods, as well as some state-of-the-art BOIQA methods.

Region-level bit allocation for rate control of 360-degree videos using cubemap projection

Featuring with more uniform sampling density in the sphere domain and less non-uniform geometric deformations in the planar domain, variants of cubemap projection (CMP) format enable the higher compression ratio in on-going 360-degree video coding standardization. Different from single-view videos, 360-degree CMP videos feature with content discontinuity combined with the abrupt change of motion vectors between some adjacent faces. However, there is few bit allocation scheme designed for rate control of video coding of CMP format. Thus, this paper proposes a region-level bit allocation scheme for rate control of interframe coding of CMP format. The proposed scheme consists of two parts. The first part is machine learning based high HEVC coding cost region detection for individual faces, where the feature descriptor of a CTU consists of the face based texture complexity, motion magnitude, motion density, and temporal coherence of motion vector. The second part is fitting function based region-level bit allocation. Different from previous work, bits are assigned to the high coding cost region and non-high coding cost region in individual faces of CMP format. Experimental results indicate that the proposed scheme achieves higher bitrate accuracy and larger BD-WS-PSNR compared with the original rate control scheme of the reference software of HEVC, HM16.16 with the 360Lib.

UniFuse: Unidirectional Fusion for 360° Panorama Depth Estimation

Learning depth from spherical panoramas is becoming a popular research topic because a panorama has a full field-of-view of the environment and provides a relatively complete description of a scene. However, applying well-studied CNNs for perspective images to the standard representation of spherical panoramas, i.e., the equirectangular projection, is suboptimal, as it becomes distorted towards the poles. Another representation is the cubemap projection, which is distortion-free but discontinued on edges and limited in the field-of-view. This letter introduces a new framework to fuse features from the two projections, unidirectionally feeding the cubemap features to the equirectangular features only at the decoding stage. Unlike the recent bidirectional fusion approach operating at both the encoding and decoding stages, our fusion scheme is much more efficient. Besides, we also designed a more effective fusion module for our fusion scheme. Experiments verify the effectiveness of our proposed fusion strategy and module, and our model achieves state-of-the-art performance on four popular datasets. Additional experiments show that our model also has the advantages of model complexity and generalization capability.

Cubemap-Based Perception-Driven Blind Quality Assessment for 360-degree Images.

image can be represented with different formats, such as the equirectangular projection (ERP) image, viewport images or spherical image, for its different processing procedures and applications. Accordingly, the 360-degree image quality assessment (360-IQA) can be performed on these different formats. However, the performance of 360-IQA with the ERP image is not equivalent with those with the viewport images or spherical image due to the over-sampling and the resulted obvious geometric distortion of ERP image. This imbalance problem brings challenge to ERP image based applications, such as 360-degree image/video compression and assessment. In this paper, we propose a new blind 360-IQA framework to handle this imbalance problem. In the proposed framework, cubemap projection (CMP) with six inter-related faces is used to realize the omnidirectional viewing of 360-degree image. A multi-distortions visual attention quality dataset for 360-degree images is firstly established as the benchmark to analyze the performance of objective 360-IQA methods. Then, the perception-driven blind 360-IQA framework is proposed based on six cubemap faces of CMP for 360-degree image, in which human attention behavior is taken into account to improve the effectiveness of the proposed framework. The cubemap quality feature subset of CMP image is first obtained, and additionally, attention feature matrices and subsets are also calculated to describe the human visual behavior. Experimental results show that the proposed framework achieves superior performances compared with state-of-the-art IQA methods, and the cross dataset validation also verifies the effectiveness of the proposed framework. In addition, the proposed framework can also be combined with new quality feature extraction method to further improve the performance of 360-IQA. All of these demonstrate that the proposed framework is effective in 360-IQA and has a good potential for future applications.

Blind quality assessment of omnidirectional videos using spatio-temporal convolutional neural networks

Panoramic videos are different from 2D videos in the spherical viewing mode. Normally, 2D videos are viewed by consumers on flat screen televisions, while panoramic videos need specific head-mounted displays to create virtual reality experience. Therefore, it is meaningful to study a quality evaluation model suitable for panoramic videos, which will bring a better quality of experience in the field of visual consumption. Unfortunately, to the best of our knowledge, there are few omnidirectional video quality assessments especially in no-reference quality evaluation modes. Besides, the existing 2D quality evaluation methods cannot well evaluate panoramic content quality. Based on above motivation and the vacancy of related research, we propose a blind deep learning-driven quality evaluation framework for panoramic videos to address the correlative issues. From the perspective of the viewport format, we eliminate the sampling distortion of panoramic videos in equirectangular projection (ERP) format via cubemap projection (CMP) format projection. In addition, we use a two-stream convolutional network to fully extract the intra-frame and inter-frame (defined as spatio-temporal) information for more comprehensive modeling of panoramic video features. Experimental results show that our method is superior to the existing full-reference and no-reference methods in solving the task of panoramic video quality assessment.

CMP-based saliency model for stereoscopic omnidirectional images

Although many saliency models for images had been proposed, the saliency model for stereoscopic omnidirectional image (SOI) is still a very important but not yet deeply studied topic. In addition, one of main disadvantages of the existing saliency models is difficult to simulate the characteristics of a wide field of view (FOV) of SOIs. To solve the problems, this paper proposes a novel cube map projection (CMP) based saliency model for SOIs. Considering that the equi-rectangular projection (ERP) representation of SOI will result in obvious stretch distortion, and what the human eyes view are the viewport images by using head mounted display (HMD), we construct the saliency model of the SOI in its less shape distorted CMP plane. First, the SOI with ERP format is converted into the CMP format. In order to reduce the boundary effect between the faces in CMP plane, its four horizontal faces are moved to obtain the horizontal global face and the horizontal local faces for which the corresponding saliency maps are subsequently computed by combining color distance and depth distance in the graph model. Then, to establish the correlations of these faces, the horizontal face saliency map is calculated by weighting the horizontal global face saliency map to the horizontal local face saliency map. Meanwhile, the vertical face saliency map is defined and obtained. Finally, the horizontal face saliency map and vertical face saliency map are spliced and re-projected to ERP plane to obtain the final saliency map of the ERP-mapped SOI. The experiments on the public ODI database are performed to compare the proposed methods with the state-of-the-art methods, and the results show that the proposed method achieves better performance of estimating SOI saliency map in terms of six well-known quantitative metrics.

Subjective and Viewport-based Objective Quality Assessment of 360-degree videos

Visual distortions in processed 360-degree visual content and consumed through head-mounted displays (HMDs) are perceived very differently when compared to traditional 2D content. To better understand how compression-related artifacts affect the overall perceived quality of 360-degree videos, this paper presents a subjective quality assessment study and analyzes the performance of objective metrics to correlate with the gathered subjective scores. In contrast to previous related work, the proposed study focuses on the equiangular cubemap projection and includes specific visual distortions (blur, blockiness, H.264 compression, and cubemap seams) on both monoscopic and stereoscopic sequences. The objective metrics performance analysis is based on metrics computed in both the projection domain and the viewports, which is closer to what the user sees. The results show that overall objective metrics computed on viewports are more correlated with the subjective scores in our dataset than the same metrics computed in the projection domain. Moreover, the proposed dataset and objective metrics analysis serve as a benchmark for the development of new perception-optimized quality assessment algorithms for 360-degree videos, which is still a largely open research problem.

Video Coding Optimization for Virtual Reality 360-Degree Source

To provide excellent visual experience for customers, virtual reality (VR) sources require higher resolutions and better visual quality than traditional picture sequences. The content of a VR video can be mapped into a sphere by playing devices to present a 360° scene, which is usually called VR360 in industrial community. The most popular formats for VR360 sources are the equirectangular projection (ERP) and the cubemap projection (CMP). Both ERP and CMP pictures can be effectively projected to a virtual three-dimensional spherical surface for rendering. It brings a new challenge to the compression of VR video sources, which is how to reallocate proper bit-rate to match mainstream projection formats. The most intuitive way to deal with this challenge is to empirically assign a fixed quantization parameter (QP) to each coding unit according to its position, which evidently lacks precision, rationality, and thus, degrades coding performance. This research proposes a new entropy equilibrium optimization (EEO) methodology to enhance the coding performance of VR360 videos. Specifically, we develop a spherical bit-rate equalization strategy to obtain a block-level Lagrangian multiplier (lambda, λ) for the rate-distortion optimization process in video coding. The appropriate QP value for each block is then dynamically determined in accordance with its λ. Based on our EEO methodology, we develop two algorithms, EEOA-ERP and EEOA-CMP, to enhance compression efficiency for the ERP and CMP pictures, respectively. Experimental results demonstrate that both algorithms achieve significant BD-Rate savings and outperform the HM16.17 platform for all-intra (AI), low-delay (LD) and random-access (RA) configurations, respectively. Concretely, compared with the state-of-the-art algorithm WSU-ERP, the proposed EEOA-ERP achieves BD-Rate saving of 0.37% in LD configuration. Furthermore, the proposed EEOA-CMP gains 2.6% on objective quality in RA configuration when compared with the HM16.17 VR CMP under the common test condition.

Discrete Spherical Image Representation for CNN-Based Inclination Estimation

How an image is represented as the input of a convolutional neural network (CNN) is important because this input directly influences the performance of the CNN. In this paper, we investigate the representation of spherical images by focusing on the inclination estimation of a spherical camera. Unlike other approaches to CNN-based inclination estimation, a spherical image is represented as a geodesic-division-based discrete spherical image (DSI) that is obtained by sampling a sphere as uniformly as possible. The input of the CNN is a single image that consists of five parallelograms flattened from a regular icosahedron. To demonstrate the advantage of the proposed method, comparative experiments are conducted with two other spherical image representations, namely, equirectangular projection (ERP) and cubemap projection (CMP). The experimental results show that the proposed method using a geodesic-division-based discrete spherical image as the CNN input obtains the best performance-better than that of the cubemap and far superior to that of the equirectangular image. The effect of the image representations used becomes more significant as the relative inclination decreases. Moreover, comparative experiments are conducted using the state-of-the-art methods for spherical camera inclination compensation to further illustrate the superiority of the DSI representation. Consequently, the proposed method provides an important reference for the development of CNNs intended for spherical images.

Spherical Lanczos Interpolation in Planar Projection or Format Conversions of Panoramic Videos

Panoramic videos provide users with an amazing experience with immersive and interactive viewing, and are now gaining global popularity. Since panoramic videos are spherical in nature while effectively processed in planar projection formats in applications, it is crucial to implement sphere-to-plane projection, where a good interpolation algorithm is of great importance regarding generating high quality planar perspective videos. Moreover, different planar projection formats adapt to different applications according to their own characteristics. It is common to convert one projection format to another, where interpolation algorithms also make a difference on performance. In this paper, an interpolation algorithm named spherical Lanczos (SLAN) is proposed to achieve advanced performance in sphere-to-plane projection or planar projection format conversions of panoramic videos. In SLAN, we fully consider panoramic videos are naturally spherical. Reference pixels are selected carefully, and the weights are calculated correctly. Experimental results demonstrate that the SLAN interpolation algorithm is beneficial for the whole processing chain of panoramic videos both with lossless compression coding and with loss compression coding in terms of the increase in various end-to-end PSNR and the decrease in BD_rate. The cubic projection (CBP) and adjusted cubemap projection (ACP) are also evaluated to illustrate that the proposed algorithm is applicable for all kinds of planar projection format due to their inevitable deformation.

Эффективные методы и алгоритмы синтеза видео 360 градусов на основе кубической проекции виртуального окружения

The paper deals with the task of creation and playback of panoramic video with 360-degree overview, which allows the researcher to be immersed in virtual medium outside parent virtual environment system (VES). To solve the task, the extension of the method is proposed, in which resolution is determined taking into account viewer camera field of view and screen resolution (Adequate Cubemap Projection, ACMP). The paper studies the influence of the camera orientation inside the cube on the cubemap pixel / screen pixel ratio determining panorama visualization quality. Based on this, a method to calculate resolution for high-quality panorama visualization for all possible camera orientations is proposed. The paper considers an efficient method and algorithm to create ACMP-video on the GPU using render-to-texture technology, which allow to synthesize panoramas with constant orientation or bound to the observer's view direction. In the research efficient methods and algorithms to play ACMP-video are also proposed, which are based on the visualization of visible cube faces and adaptive frame buffering. The obtained methods and algorithms are implemented in ACMP-video synthesis program complex (С++, OpenGL, FFmpeg) including frame capture module (embeddable into VES) and the player. The developed solution was tested in system «Virtual Earth» designed for training to observe Earth objects from the International Space Station (ISS). Using the capture module, an ACMP-video of the flight along the ISS orbit track was created. When playing this video, the trainee flies in orbit above virtual 3D surface of the Earth and can explore it by means of camera rotation. Testing of the complex confirmed the adequacy of the developed methods and algorithms to the task. The obtained scientific and practical results expand the capabilities and scope of application of VES, scientific visualization systems, video simulators and virtual laboratories; provide effective exchange of experience between researchers, etc.

Learning Compressible 360 ° Video Isomers.

Standard video encoders developed for conventional narrow field-of-view video are widely applied to 360° video as well, with reasonable results. However, while this approach commits arbitrarily to a projection of the spherical frames, we observe that some orientations of a 360° video, once projected, are more compressible than others. We introduce an approach to predict the sphere rotation that will yield the maximal compression rate. Given video clips in their original encoding, a convolutional neural network learns the association between a clip's visual content and its compressibility at different rotations of a cubemap projection. Given a novel video, our learning-based approach efficiently infers the most compressible direction in one shot, without repeated rendering and compression of the source video. We validate our idea on thousands of video clips and multiple popular video codecs. The results show that this untapped dimension of 360° compression has substantial potential-"good" rotations are typically 8-18 percent more compressible than bad ones, and our learning approach can predict them reliably 78 percent of the time.

A Generalized Cubemap for Encoding 360° VR Videos using Polynomial Approximation

Abstract360° VR videos provide users with an immersive visual experience. To encode 360° VR videos, spherical pixels must be mapped onto a two‐dimensional domain to take advantage of the existing video encoding and storage standards. In VR industry, standard cubemap projection is the most widely used projection method for encoding 360° VR videos. However, it exhibits pixel density variation at different regions due to projection distortion. We present a generalized algorithm to improve the efficiency of cubemap projection using polynomial approximation. In our algorithm, standard cubemap projection can be regarded as a special form with 1st‐order polynomial. Our experiments show that the generalized cubemap projection can significantly reduce the projection distortion using higher order polynomials. As a result, pixel distribution can be well balanced in the resulting 360° VR videos. We use PSNR, S‐PSNR and CPP‐PSNR to evaluate the visual quality and the experimental results demonstrate promising performance improvement against standard cubemap projection and Google's equi‐angular cubemap.