Omnidirectional Images Research Articles

Convolutional Neural Networks for Omnidirectional Image Quality Assessment: A Benchmark

In this paper, we conduct an extensive study on the use of pre-trained convolutional neural networks (CNNs) for omnidirectional image quality assessment (IQA). To cope with the lack of available IQA databases, transfer learning from seven pre-trained CNN models is investigated over retraining on standard 2D databases. In addition, we explore the influence of various image representations and training strategies on the model’s performance. A comparison of the use of projected versus radial content, and multichannel CNN versus patch-wise training is also covered. The experimental results on two publicly available databases are used to draw conclusions about which strategy best fits the visual quality prediction and at which computational cost. The analysis shows that retraining CNN models on 2D IQA databases improves the prediction accuracy. The latter and the required computational time are found to be significantly affected by the training strategy. Cross-database evaluations demonstrate that the nature and variety of the content impact the generalization ability of the models. Finally, we show that conclusions coming from other image processing communities may not hold for IQA. The provided discussion shall provide insights and recommendations when using pre-trained CNNs for omnidirectional IQA.

GNSS Urban Positioning with Vision-Aided NLOS Identification

The global navigation satellite system (GNSS) has played an important role in a broad range of consumer and industrial applications. In particular, cities have become GNSS major application scenarios; however, GNSS signals suffer from blocking, reflection and attenuation in harsh urban environments, resulting in diverse received signals, e.g., non-line-of-sight (NLOS) and multipath signals. NLOS signals often cause severe deterioration in positioning, navigation, and timing (PNT) solutions, which should be identified and excluded. In this paper, we propose a vision-aided NLOS identification method to augment GNSS urban positioning. A skyward omnidirectional camera is installed on a GNSS antenna to collect omnidirectional images of the sky region. After being rectified, these images are processed for sky region segmentation, which is improved by leveraging gradient information and energy function optimization. Image morphology processing is further employed to smooth slender boundaries. After sky region segmentation, the satellites are projected onto the omnidirectional image, from which NLOS satellites are identified. Finally, the identified NLOS satellites are excluded from GNSS PNT estimation, promoting accuracy and stability. Practical test results show that the proposed sky region segmentation module achieves over 96% accuracy, and that completely accurate NLOS identification is achieved for the experimental images. We validate the performance of our method on public datasets. Compared with the raw measurements without screening, the vision-aided NLOS identification method enables improvements of 60.3%, 12.4% and 63.3% in the E, N, and U directions, respectively, as well as an improvement of 58.5% in 3D accuracy.

An intelligent ultrasonic guided wave deep imaging detection method considering the propagation characteristics of each direction

Macroscopic damage in multilayer anisotropic structures is usually formed by the rapid development of material micro-damage. The existing micro-damage imaging detection technology does not consider the difference of wave velocity in all directions of the detection structure, especially not directly taking the wave velocities in different direction into the imaging counting process, the micro-damage imaging detection, and the false detection rate and missed detection rate are high. In this paper, a deep learning imaging detection method considering velocity in all directions is proposed and verified on a carbon fiber anticorrosive coating structure of a shaftless ring propeller drive system. Firstly, the problem that the elliptical damage path cannot be determined in the anisotropic structure is analyzed, and the omnidirectional velocity of the carbon fibre reinforced plastics structure is obtained through simulation analysis. A new omni-directional imaging method was proposed, which discretized the monitored objects and acquired the damage index through deep learning network. The damage propagation time of the reference point was compared with that of the actual damage point to determine the damage probability of the structure. The experimental results show that the omni-directional imaging method can accurately and intuitively display the damage information of anisotropic structures.

VSOIQE: A Novel Viewport-Based Stitched 360° Omnidirectional Image Quality Evaluator

With the rapid development of virtual reality (VR), 360° omnidirectional images and videos have drawn wide attention. However, the quality assessment of 360° omnidirectional images is a challenging task, especially when the panoramic image contains multiple stitching distortions. We propose a viewport-based stitched 360° omnidirectional image quality evaluator (VSOIQE), by first extracting the features of salient and stitching viewports, and then inferring the overall perceptual quality via multiple linear regression (MLR). Comprehensive image attributes including edge, color, shape and information entropy are considered in the framework. Experimental results on two benchmark databases demonstrate the superiority of the proposed metric over both the state-of-the-art quality models designed for 2D images and the quality models developed for 360° omnidirectional images.

An effective feature detection approach for image stitching of near-uniform scenes

The first step to accomplish panoramic or omnidirectional image stitching often rely on how well the feature detector extract sufficient number of matchable corresponding interest points between images. This step remains a challenging problem to many renowned feature detectors, especially when the image content is fairly near-uniform. The awful impacts are often reflected in obvious misaligned and projective distortion. To address this problem, we propose an effective feature detection method based on Log-Gabor feature transform. Instead of depending on image intensity for feature point detection, the proposed method utilizes log-Gabor function to construct a number of scaled and oriented log-Gabor scale spaces from which more distinctive and matchable features can be extracted effectively. In addition, we also introduce new evaluation metrics which are more relevant for evaluating the performance of image stitching. Our method has a better feature matching performance than other state-of-the-art feature extraction methods, particularly for near-uniform scenes.

Decoupled dynamic group equivariant filter for saliency prediction on omnidirectional image

Current saliency prediction models based on convolutional neural networks (CNNs) achieve solid improvement in predicting human attention on omnidirectional image (ODI). However, these models that employ standard convolution have two main shortcomings: content-agnostic and computation-intensive. To address these two shortcomings, we propose a decoupled dynamic group equivariant filter (DDGF). Specifically, inspired by the attention mechanism that adopts light-weight branches for estimating spatial and channel attention, we decouple group equivariant convolution (i.e. p4 convolution) into spatial and channel dynamic group equivariant filters. Such a design not only makes p4 convolution filter adaptive to ODI content, but also considerably reduces computational cost. To our best knowledge, the DDGF is the first decoupled dynamic convolution filter that applied to the task of saliency prediction. Meanwhile, we observe that it is effective and efficient when replacing standard group equivariant convolution with DDGF in ODI saliency prediction. Experimental results show that the proposed DDGF can achieve superior performance in comparison with other state-of-the-art methods. Additionally, we conduct ablation experiments to verify the effectiveness of each component of the proposed DDGF.

Parallel Lines for Calibration of Non-Central Conical Catadioptric Cameras

In this paper we propose a new calibration method for non-central catadioptric cameras that use a conical mirror. This method consists of using parallel lines, extracted from a single omnidirectional image, instead of using the typical checkerboard to obtain the calibration parameters of the system.

MEMS mirror based omnidirectional scanning for lidar optical systems

Lidar 3D vision system is currently one of the most effective technical means for high-precision perception of three-dimensional image information of the surrounding environment. Microelectron-mechanical systems (MEMS) micromirror lidar has advantages of miniaturization, fast scanning speed, and relatively low price. The MEMS micromirror scanning method has become the best choice for laser scanning 3D image sensors. However, due to the small size of the MEMS micromirror, it can only scan a small angle range, which limits its industrial applications. In this paper, combined with MEMS micro-mirror spiral scanning form, we design a new type of omnidirectional optical system that can realize MEMS lidar omnidirectional scanning imaging. The overall optical system is composed of the catadioptric prism, the MEMS micromirror, and the compensation prism. The overall system can achieve 360° omnidirectional scanning imaging with a horizontal resolution of 0.1°; a vertical field view with a 0.5° resolution. The design results are analyzed by the optical design software (Zemax). The point diagram shows that the emitted light spots are circular and evenly distributed. The divergence angle is less than 1mrad, which can well meet the actual needs and solve the problems of solid-state, miniaturization, and omnidirectional scanning imaging of lidar.

Multi-Angle Projection Based Blind Omnidirectional Image Quality Assessment

Most of the existing blind omnidirectional image quality assessment (BOIQA) methods are based on data-driven approach where the end-to-end neural network or deep learning tools are mainly used for feature extraction. However, it usually lacks interpretability and is difficult to discover the perceptual mechanism behind. In this paper, from the perspective of perception modeling, we propose a novel multi-angle projection based BOIQA (MP-BOIQA) method. Considering the omnibearing and near eye display characteristics with head mounted display, multiple color cubemap projection images with respect to different viewpoints are grouped as the color omnidirectional distortion (COD) units so as to simulate the user’s viewing behavior in subjective quality assessment. In the designed multi-angle projection based feature extractor, tensor decomposition is implemented on each COD unit for dimensionality reduction, and piecewise exponential fitting is used to get the distribution of mean subtracted contrast normalized coefficients of the unit’s feature matrices in tensor domain. Finally, the extracted features are pooled with random forest. The experimental results on three omnidirectional image quality datasets show that the MP-BOIQA method can deliver highly competitive performance compared with some representative full-reference quality assessment methods, as well as some state-of-the-art BOIQA methods.

Perceptual Quality Assessment of Omnidirectional Images

Omnidirectional images, also called 360◦images, have attracted extensive attention in recent years, due to the rapid development of virtual reality (VR) technologies. During omnidirectional image processing including capture, transmission, consumption, and so on, measuring the perceptual quality of omnidirectional images is highly desired, since it plays a great role in guaranteeing the immersive quality of experience (IQoE). In this paper, we conduct a comprehensive study on the perceptual quality of omnidirectional images from both subjective and objective perspectives. Specifically, we construct the largest so far subjective omnidirectional image quality database, where we consider several key influential elements, i.e., realistic non-uniform distortion, viewing condition, and viewing behavior, from the user view. In addition to subjective quality scores, we also record head and eye movement data. Besides, we make the first attempt by using the proposed database to train a convolutional neural network (CNN) for blind omnidirectional image quality assessment. To be consistent with the human viewing behavior in the VR device, we extract viewports from each omnidirectional image and incorporate the user viewing conditions naturally in the proposed model. The proposed model is composed of two parts, including a multi-scale CNN-based feature extraction module and a perceptual quality prediction module. The feature extraction module is used to incorporate the multi-scale features, and the perceptual quality prediction module is designed to regress them to perceived quality scores. The experimental results on our database verify that the proposed model achieves the competing performance compared with the state-of-the-art methods.

A novel method for extracting geometric parameter information of buildings based on CSAR images

ABSTRACT Compared with the traditional linear synthetic aperture radar (LSAR), circular SAR (CSAR) system can observe 360° of the observation area. In turn, more abundant scattering information in all directions of the observation area can be obtained, making it easier to observe the target information. Based on the observation advantages of CSAR, a new method for extracting geometric parameters of buildings is proposed in this paper. In this method, the scattering features of buildings on CSAR sub-aperture images and full-aperture images are extracted, and the corresponding relationship between the scattering features and building geometric parameters is deduced, and then the building geometric parameters are calculated. Through the above corresponding relationship, the geometric parameters of the building are calculated. At the same time, in this method, the advantages of CSAR omnidirectional imaging are used to extract the geometric parameters of buildings from multiple sub-aperture images. This method realizes the high-precision extraction of building geometric parameters by CSAR imaging for the first time. Finally, the experimental verification is carried out by introducing the measured data of the Ku-band autonomously measured, which confirms the effectiveness and accuracy of the proposed method.

Multiscale Brain-Like Neural Network for Saliency Prediction on Omnidirectional Images

Current top-performing saliency prediction methods of omnidirectional images (ODIs) depend on deep feedforward convolutional neural networks (CNNs), benefiting from their powerful multiscale representation ability. Although these methods adopt deep feedforward CNNs to achieve superb performance in saliency prediction task, they have the following limitations: 1) these deep feedforward CNNs are difficult to map to ventral stream structure of the brain visual system due to their vast number of layers and missing biologically important connections, such as recurrence and 2) most deep feedforward CNNs represent the multiscale features in a layerwise manner. To tackle these issues, models that could learn multiscale features yet share the similarities with human brain are needed. In this article, we propose a novel multiscale brain-like network (MBN) model to predict saliency of head fixations on ODIs. Specifically, our proposed model consists of two major modules: 1) a brain-like CORnet-S module and 2) a multiscale feature extraction module. The CORnet-S module is a lightweight backbone network with four anatomically mapped areas (V1, V2, V4, and IT) and it can simulate the visual processing mechanism of ventral visual stream in the human brain. The multiscale feature extraction module is inspired by the multiscale brain structure, which represents multiscale features at a granular level and increases the range of receptive fields for each network layer. Extensive experiments and ablation studies conducted on two major benchmarks demonstrate the superiority of the proposed MBN model over the state-of-the-art methods.

Monocular and Binocular Interactions Oriented Deformable Convolutional Networks for Blind Quality Assessment of Stereoscopic Omnidirectional Images

Stereoscopic omnidirectional content, as a novel visual media, has drawn wide attention in recent years due to its ability in providing strong immersive experience. Since Stereoscopic Omnidirectional Images (SOIs) involve the properties from panoramic and stereoscopic visual perception, it is very challenging to establish an efficient and effective visual quality evaluation model for SOIs. To better measure the user’s experience in virtual reality, we put forward a novel deep learning framework to assess the quality of SOIs in this paper. Firstly, the deformable convolutions instead of standard convolutions are adopted to ensure the invariant receptive fields of convolutional kernels on Equi-Rectangular Projection (ERP). Secondly, according to the stereoscopic property, we use binocular-difference information and a coarse-to-fine mechanism to construct the binocular feature extraction network. Thirdly, a three-channel network involving left-view, right-view and binocular-difference channels is presented to simulate the process of monocular and binocular interactions, in which independent quality labels are provided for each channel to reflect the individual effect of monocular and binocular visions on the whole visual quality. Finally, experimental results on two available benchmark databases demonstrate the superiority of the proposed metric over the state-of-the-art blind quality assessment models in predicting the quality of SOIs. Moreover, our model is efficient in computational cost as the feature extraction is directly applied on ERP images.

Training, Optimization and Validation of a CNN for Room Retrieval and Description of Omnidirectional Images

The aim of this paper is to achieve the optimal hyperparameters setup of a convolutional neural network (CNN) to address the localization of a mobile robot. The localization problem is solved with a hierarchical approach by using omnidirectional images as provided by a catadioptric visual sensor, with no panoramic conversion. In this way, we propose adapting and re-training AlexNet with a double purpose. First, to perform the rough localization step by means of a room retrieval task. Second, to carry out the fine localization step within the retrieved room, in which the CNN is used to obtain a holistic descriptor that is compared with the visual model of the retrieved room by means of a nearest neighbour search. To achieve this, a CNN has been adapted and re-trained to address both the room retrieval problem and the obtention of holistic descriptors from raw omnidirectional images. The novelty of this work is the use of a data augmentation technique and Bayesian optimization to address the training process robustly. As shown in the present paper, these tools have proven to be an efficient and robust solution to the localization problem even with substantial changes of the lighting conditions of the target environment.

No-Reference Quality Assessment for 360-Degree Images by Analysis of Multifrequency Information and Local-Global Naturalness

360-degree/omnidirectional images (OIs) have received remarkable attention due to the increasing applications of virtual reality (VR). Compared to conventional 2D images, OIs can provide more immersive experiences to consumers, benefiting from the higher resolution and plentiful field of views (FoVs). Moreover, observing OIs is usually in a head-mounted display (HMD) without references. Therefore, an efficient blind quality assessment method, which is specifically designed for 360-degree images, is urgently desired. In this paper, motivated by the characteristics of the human visual system (HVS) and the viewing process of VR visual content, we propose a novel and effective no-reference omnidirectional image quality assessment (NR OIQA) algorithm by MultiFrequency Information and Local-Global Naturalness (MFILGN). Specifically, inspired by the frequency-dependent property of the visual cortex, we first decompose the projected equirectangular projection (ERP) maps into wavelet subbands by using discrete Haar wavelet transform (DHWT). Then, the entropy intensities of low-frequency and high-frequency subbands are exploited to measure the multifrequency information of OIs. In addition to considering the global naturalness of ERP maps, owing to the browsed FoVs, we extract the natural scene statistics (NSS) features from each viewport image as the measure of local naturalness. With the proposed multifrequency information measurement and local-global naturalness measurement, we utilize support vector regression (SVR) as the final image quality regressor to train the quality evaluation model from visual quality-related features to human ratings. To our knowledge, the proposed model is the first no-reference quality assessment method for 360-degree images that combines multifrequency information and image naturalness. Experimental results on two publicly available OIQA databases demonstrate that our proposed MFILGN outperforms state-of-the-art full-reference (FR) and NR approaches.

A comparative study of semantic segmentation of omnidirectional images from a motorcycle perspective

The semantic segmentation of omnidirectional urban driving images is a research topic that has increasingly attracted the attention of researchers, because the use of such images in driving scenes is highly relevant. However, the case of motorized two-wheelers has not been treated yet. Since the dynamics of these vehicles are very different from those of cars, we focus our study on images acquired using a motorcycle. This paper provides a thorough comparative study to show how different deep learning approaches handle omnidirectional images with different representations, including perspective, equirectangular, spherical, and fisheye, and presents the best solution to segment road scene omnidirectional images. We use in this study real perspective images, and synthetic perspective, fisheye and equirectangular images, simulated fisheye images, as well as a test set of real fisheye images. By analyzing both qualitative and quantitative results, the conclusions of this study are multiple, as it helps understand how the networks learn to deal with omnidirectional distortions. Our main findings are that models with planar convolutions give better results than the ones with spherical convolutions, and that models trained on omnidirectional representations transfer better to standard perspective images than vice versa.

People detection with omnidirectional cameras using a spatial grid of deep learning foveatic classifiers

A novel deep-learning people detection algorithm using omnidirectional cameras is presented, which only requires point-based annotations, unlike most of the prominent works that require bounding box annotations. Thus, the effort of manually annotating the needed training databases is significantly reduced, allowing a faster system deployment. The algorithm is based on a novel deep neural network architecture that implements the concept of Grid of Spatial-Aware Classifiers, but allowing end-to-end training that improves the performance of the whole system. The designed algorithm satisfactorily handles the severe geometric distortions of the omnidirectional images, which typically degrades the performance of state-of-the-art detectors, without requiring any camera calibration. The algorithm has been evaluated in well-known omnidirectional image databases (PIROPO, BOMNI, and MW-18Mar) and compared with several works of the state of the art.

Omnisupervised Omnidirectional Semantic Segmentation

Modern efficient Convolutional Neural Networks (CNNs) are able to perform semantic segmentation both swiftly and accurately, which covers typically separate detection tasks desired by Intelligent Vehicles (IV) in a unified way. Most of the current semantic perception frameworks are designed to work with pinhole cameras and benchmarked against public datasets with narrow Field-of-View (FoV) images. However, there is a large accuracy downgrade when a pinhole-yielded CNN is taken to omnidirectional imagery, causing it unreliable for surrounding perception. In this paper, we propose an omnisupervised learning framework for efficient CNNs, which bridges multiple heterogeneous data sources that are already available in the community, bypassing the labor-intensive process to have manually annotated panoramas, while improving their reliability in unseen omnidirectional domains. Being omnisupervised, the efficient CNN exploits both labeled pinhole images and unlabeled panoramas. The framework is based on our specialized ensemble method that considers the wide-angle and wrap-around features of omnidirectional images, to automatically generate panoramic labels for data distillation. A comprehensive variety of experiments demonstrates that the proposed solution helps to attain significant generalizability gains in panoramic imagery domains. Our approach outperforms state-of-the-art efficient segmenters on highly unconstrained IDD20K and PASS datasets.

An Elastic Frame Rate Up-Conversion for Sequential Omnidirectional Images

A distinct innovation in the development of frame rate up-conversion for an omnidirectional video is presented in this paper. The innovation allows an omnidirectional video to decide the number of omnidirectional frames that the main algorithm can create based on the characteristic of the motion of objects in the omnidirectional video itself. This elastic omnidirectional frame production is achievable since the main algorithm works based on an optical flow method with a self-improvement mechanism. The optical flow concept has proven to be adaptable greatly with the character of such a video. The experiment results, in which ten omnidirectional videos were involved, show that the algorithm's exertion and adaptability named the elastic frame rate up-conversion (E-FRUC) are incredible. This achievement is confirmed by both the excellent visual quality and the high score of the PSNR (33 to 59 dB) of the reconstructed omnidirectional frame (RF). The E-FRUC can generate at least one RF from two consecutive omnidirectional images, although these images are synthetically designed with an ideal motion. The more complex the motion objects condition inside the omnidirectional video is, the more RFs the E-FRUC can create. By using the E-FRUC, the continuity of motion of moving objects can be preserved. Therefore it can be useful to maintain the frame rate of incomplete omnidirectional video frames when such video is played back. Such a condition usually happens when the omnidirectional video is transmitted through error-prone telecommunication networks.

SalyPath360: Saliency and scanpath prediction framework for omnidirectional images

This paper introduces a new framework to predict visual attention of omnidirectional images. The key setup of our architecture is the simultaneous prediction of the saliency map and a corresponding scanpath for a given stimulus. The framework implements a fully encoder-decoder convolutional neural network augmented by an attention module to generate representative saliency maps. In addition, an auxiliary network is employed to generate probable viewport center fixation points through the $SoftArgMax$ function. The latter allows to derive fixation points from feature maps. To take advantage of the scanpath prediction, an adaptive joint probability distribution model is then applied to construct the final unbiased saliency map by leveraging the encoder decoder-based saliency map and the scanpath-based saliency heatmap. The proposed framework was evaluated in terms of saliency and scanpath prediction, and the results were compared to state-of-the-art methods on Salient360! dataset. The results showed the relevance of our framework and the benefits of such architecture for further omnidirectional visual attention prediction tasks.