Omnidirectional Images Research Articles

Geometric-driven structure recovery from a single omnidirectional image based on planar depth map learning

Geometric-driven structure recovery from a single omnidirectional image based on planar depth map learning

Lightweight Distortion-Aware Network for Salient Object Detection in Omnidirectional Images

Compared with 2D image salient object detection (SOD), SOD in omnidirectional images (or 360° images) usually suffers from geometric distortion. Although existing omnidirectional image SOD (ODI-SOD) methods have improved the detection accuracy obviously, their application may be cumbersome in real scenes due to their high computational cost. To avoid distortion and reduce the computational cost simultaneously in ODI-SOD, we propose a novel lightweight distortion-aware network, named LDNet, in this paper. First, to extract features with less distortion from ODIs, we integrate the distortion-aware convolution and depth-wise separable convolution (DSConv) into distortion-aware DSConv (DDSConv) and replace the regular convolutions in the last two blocks of the ResNet-18 with DDSConvs to obtain our lightweight backbone network (LD-ResNet-18). To enhance spatial information in each channel of the extracted features at each level comprehensively, then, we propose a lightweight distortion-aware channel-wise enhancement (DCE) module (only 0.05M parameters) including DDSConvs with various dilation rates, channel shuffle operation and attention mechanism, and employ a high-to-low dense connection structure to modulate the enhanced multi-level features. Besides, we design a distortion-aware self-correlation (DSC) module (only 0.02M parameters) for mining the contextual dependency of the features via a coarse-fine strategy, and the correlated features are refined by DCE modules and integrated by another dense connection structure. The final saliency map is predicted from the densely integrated features. Compared with 12 state-of-the-art methods on two public datasets, our lightweight LDNet achieves competitive or even better performance with only 2.9M parameters and 3.4G FLOPs, which balances the efficiency and performance.

Multifaceted perception-based blind omnidirectional image quality assessment

Multifaceted perception-based blind omnidirectional image quality assessment

Development of IoT-Based Real-Time Fire Detection System Using Raspberry Pi and Fisheye Camera

In this study, an IoT-based fire detection system was developed to detect and prevent damage from forest fires at an early stage. In Korea, forest fires spread quickly due to the dry climate and winds in spring and autumn, so quick detection and prevention is necessary. To quickly detect and prevent forest fires that occur periodically, a real-time fire detection system was developed by combining a Raspberry Pi and a fisheye camera. A lens with a 220° angle of view was installed, and an image analysis algorithm was implemented using the OpenCV library. The location of the fire was estimated by calculating the polar coordinates of the omnidirectional images. Using the Wi-Fi communication function of the Raspberry Pi, the acquired continuous images were transmitted to the Firebase database, and the images were analyzed to identify the movement path of the forest fire. The developed system was applied to a mountainous area near the Samcheok Campus of Kangwon National University. As a result of the experiment, when the location of points about 25.9 m (average) away from the observation point was predicted, the positional error was analyzed to be about 1.1 m. If the system is improved in the future, it is expected that it will be able to contribute to the early prevention of forest fires with fast and accurate responses.

Optimizing Appearance-Based Localization with Catadioptric Cameras: Small-Footprint Models for Real-Time Inference on Edge Devices.

This paper considers the task of appearance-based localization: visual place recognition from omnidirectional images obtained from catadioptric cameras. The focus is on designing an efficient neural network architecture that accurately and reliably recognizes indoor scenes on distorted images from a catadioptric camera, even in self-similar environments with few discernible features. As the target application is the global localization of a low-cost service mobile robot, the proposed solutions are optimized toward being small-footprint models that provide real-time inference on edge devices, such as Nvidia Jetson. We compare several design choices for the neural network-based architecture of the localization system and then demonstrate that the best results are achieved with embeddings (global descriptors) yielded by exploiting transfer learning and fine tuning on a limited number of catadioptric images. We test our solutions on two small-scale datasets collected using different catadioptric cameras in the same office building. Next, we compare the performance of our system to state-of-the-art visual place recognition systems on the publicly available COLD Freiburg and Saarbrücken datasets that contain images collected under different lighting conditions. Our system compares favourably to the competitors both in terms of the accuracy of place recognition and the inference time, providing a cost- and energy-efficient means of appearance-based localization for an indoor service robot.

3D underwater acoustic Luneburg lens based on gradient face-centered-cubic phononic crystals

A Luneburg lens is a gradient refractive index lens that can focus plane waves on a point at the perimeter without aberration. Three-dimensional (3D) Luneburg lens for airborne sound has been well investigated in recent years. However, constructing a 3D Luneburg lens for underwater sound is a challenging task due to the difficulties in the designing and fabricating of the desired isotropic underwater acoustic materials. This work presents the practical implementation of a 3D Luneburg lens for underwater sound. Such a 3D Luneburg lens is designed based on 3D gradient face-centered-cubic phononic crystals, which have quasi-isotropic refractive index patterns and can be fabricated with photosensitive resin by 3D printing. The experimental results show that the lens can realize the omnidirectional imaging of underwater sound from 30 to 38 kHz. This 3D underwater acoustic Luneburg lens may prompt the potential applications in underwater acoustic wide-angle retroreflectors, sonars, and biomedical imaging devices.

A no-reference panoramic image quality assessment with hierarchical perception and color features

The ultimate goal of no-reference omnidirectional image quality assessment (NR-OIQA) is to design a comprehensive perception method that can accurately assess the quality of damaged omnidirectional images without prior knowledge. However, most existing studies cannot attain credible accuracy because of lacking a non-neuroscience-based or non-biology-based model. Inspired by this, an original visual perception-based and neuroscience-based OIQA model by considering the hierarchical perception features of the HVS, which includes specific information, local saliency information, global information, and color information which is often ignored by researchers is proposed in this work. According to the hierarchical process in neuroscience, the high-frequency co-occurrence matrix (HFCM)-based and variance-based specific features are applied to perceive details that are first distorted in the frequency domain. The entropy-based combination of the paranormal saliency map(PSM) and superpixel segmentation with the simple linear iterative clustering (SLIC) algorithm is employed to emphasize the rich quality-aware local saliency information. The global panoramic statistical (GPS) model is utilized to express global semantics distortion as the high-level feature. Visual-aware color texture descriptor with the cross-channel local binary pattern(CCLBP) which effectively reflects the correlation and dependency of pixels between different color channels is employed to map color information. Finally, all above features have been extracted and combined with subjective scores to assess the objective quality scores by support vector regression (SVR). Experiments express that our method has more accuracy and stronger stability on CVIQD2018 and OIQA databases.

MODE: Monocular omnidirectional depth estimation via consistent depth fusion

Monocular depth estimation has seen significant progress in recent years, especially in outdoor scenes. However, depth estimation results are not satisfying in omnidirectional images. As compared to perspective images, estimating the depth map from an omnidirectional image captured in the outdoor scene, using neural networks, has two additional challenges: (i) the depth range of outdoor images varies a lot across different scenes, making it difficult for the depth network to predict accurate depth results for training with an indoor dataset, besides the maximum distance in outdoor scenes mostly stays the same as the camera sees the sky, but depth labels in this region are entirely missing in existing datasets; (ii) a standard representation of omnidirectional images introduces spherical distortion, which causes difficulties for the vanilla network to predict accurate relative structural depth details. In this paper, we propose a novel network-MODE by giving special considerations to those challenges and designing a set of flexible modules for improving the performance of omnidirectional depth estimation. First, a consistent depth structure module is proposed to estimate a consistent depth structure map, and the predicted structural map can improve depth details. Second, to suit the characteristics of spherical sampling, we propose a strip convolution fusion module to enhance long-range dependencies. Third, rather than using a single depth decoder branch as in previous methods, we propose a semantics decoder branch to estimate sky regions in the omnidirectional image. The proposed method is validated on three widely used datasets, demonstrating the state-of-the-art performance. Moreover, the effectiveness of each module is shown through an ablation study on real-world datasets. Our code is available at https://github.com/lkku1/MODE.© 2017 Elsevier Inc. All rights reserved.

Viewport-Sphere-Branch Network for Blind Quality Assessment of Stitched 360° Omnidirectional Images

Compared with conventional images/videos, omni-directional data records rich information with higher resolution and wider Field-of-View. Moreover, the stitching distortions introduced in the panoramic content generation process make the quality assessment task more challenging. Targeting at designing an accurate and fast stitched 360° omnidirectional image quality evaluator, we propose a Viewport-Sphere-Branch Network (VSBNet) via dual-branch quality estimation. Specifically, for the viewport quality estimation, we extract distorted viewports around the stitching seams and conduct distortion rectification through a progressively complementary network to obtain pseudo-reference viewports. The qualitative and quantitative experiments validate that pseudo-reference viewports are reliable. Then, the differences between distorted and pseudo-reference viewports are quantified through transformer architecture to obtain quality scores of viewports. The introduction of pseudo-reference viewports can effectively improve the performance of the viewport quality prediction branch. To establish general scenario awareness and accurately evaluate the immersive experience, we extract feature representation through deformable convolutions to eliminate 2D-to-Sphere intrinsic sampling distortions and use multilayer perceptron to predict score of the whole sphere. The final prediction score is obtained by aggregating the quality scores from viewport and sphere branches. We evaluate the proposed VSBNet on two benchmark databases and results demonstrate that the combination of two branches can obtain more accurate results. Overall, our method is superior to existing full reference and no reference models designed for conventional images and 360° omnidirectional images.

Omnidirectional visual computing: Foundations, challenges, and applications

Omnidirectional images and videos are widespread due to the popularization of devices for capturing and visualization. Unlike pinhole-based imagery, omnidirectional media lie on the surface of a sphere, have a 360° × 180°field of view, and store the light intensities from an entire environment. Notably, applications involving immersive augmented, mixed, and virtual reality experiences benefit from the 360° content. Although defined on the spherical domain, omnidirectional images are often mapped to a (multi-)planar representation, which results in distorted images and degrades the performance of most traditional visual computing algorithms designed to work on the plane. This paper reviews the spherical camera model, the most common capturing devices, and popular (multi-)planar representations of omnidirectional media. It also approaches the main challenges of omnidirectional visual computing, focusing on the deep learning paradigm, and tackles four visual computing applications that strongly explore the potential of omnidirectional imagery.

TCCL-Net: Transformer-Convolution Collaborative Learning Network for Omnidirectional Image Super-Resolution

As virtual reality and metaverse become more and more popular, the Omnidirectional Image (OI) has attracted extreme attention due to its immersive display characteristics. However, users only watch a portion of the content in a specific viewport extracted from a panoramic view, which will lead to a problem of resolution mismatch that requires High-Resolution (HR) for clear near-eye displays in viewports. Hence, it is necessary to exploit a Super Resolution (SR) solution for reconstructing Low-Resolution (LR) OIs. Different from 2D SR methods, the variation of pixel distributions along latitudes is a critical factor in designing an Omnidirectional Image Super-Resolution (OISR) scheme. In this paper, we put forward a novel end-to-end network with a Transformer and Convolution Collaborative Learning Network (TCCL-Net) for OISR. Firstly, Swin Transformer blocks and residual convolution blocks are employed to extract long-range and short-range dependencies, thereby digging into more rich and heterogeneous features from these two branches. Secondly, to better fuse these two features, cross-guided enhanced attention mechanisms are designed for bidirectional information enhancement onto both channel and spatial features. Thirdly, to alleviate nonuniformly pixel distributions across latitudes, we add an absolute positional encoding into Swin Transformer to represent patch weights at different positions and propose a tile-based panoramic reconstruction module to super-resolve various bands with different pixel sampling characteristics across latitudes. Experimental results on two available benchmark datasets demonstrate the superiority of the proposed approach over the state-of-the-art method in achieving OISR task.

Generalization of the shortest path approach for superpixel segmentation of omnidirectional images

With the growing use of image capture devices using wide angles and the need for fast and accurate image analysis in computer vision, there is a demand for dedicated under-representation methods. Most decomposition methods segment an image into a small number of irregular homogeneous regions, called superpixels. Nevertheless, these approaches are generally designed to process natural 2D planar images, i.e., captured with a 90o angle view without distortion. In this work, we present SphSPS, a new general decomposition method (for Spherical Shortest Path-based Superpixels)11Available code at: https://github.com/rgiraud/sphsps, that is dedicated to wide 360o omnidirectional or spherical images. The produced superpixels respect both the geometry of the 3D spherical acquisition space, and the boundaries of image objects. To fastly extract relevant clustering features, we generalize the shortest path approach between a pixel and a superpixel center. We demonstrate that considering the geometry of the acquisition space to compute the shortest path enables to jointly improve the segmentation accuracy and the shape regularity of superpixels. To evaluate this regularity property, we propose a generalization of a standard 2D regularity metric to the spherical space, addressing the limitations of the only existing spherical compactness measure. Finally, SphSPS is validated on reference 360o images from the PSD (Panorama Segmentation Dataset) and also on synthetic road omnidirectional images. Our method significantly outperforms both planar and spherical state-of-the-art approaches in terms of segmentation accuracy, robustness to noise and regularity, providing a very interesting tool for superpixel-based applications on 360o images.

The Development of 3D Atomic Force Microscopy with Magnetically Driven-Orthogonal Cantilever Probes

This paper presents a three-dimensional (3D)-atomic force microscopy (AFM) method based on magnetically driven (MD)-orthogonal cantilever probes (OCPs), in which two independent scanners with three degrees of freedom are used to achieve the vector tracking of a sample surface with a controllable angle. A rotating stage is integrated into the compact AFM system, which helps to achieve 360° omnidirectional imaging. The specially designed MD-OCP includes a horizontal cantilever, a vertical cantilever, and a magnetic bead that can be used for the mechanical drive in a magnetic field. The vertical cantilever, which has a protruding tip, can detect deep grooves and undercut structures. The design, simulation, fabrication, and performance analysis of the MD-OCP are described first. Then, the amplitude compensation and home positioning for 360° rotation are introduced. A comparative experiment using an AFM step grating verifies the ability of the proposed method to characterize steep sidewalls and corner details, with a 3D topography reconstruction method being used to integrate the images. The effectiveness of the proposed 3D-AFM based on the MD-OCP is further confirmed by the 3D characterization of a micro-electromechanical system (MEMS) device with microcomb structures. Finally, this technique is applied to determine the critical dimensions (CDs) of a microarray chip. The experimental results regarding the CD parameters show that, in comparison with 2D technology, from which it is difficult to obtain sidewall information, the proposed method can obtain CD information for 3D structures with high precision and thus has excellent potential for 3D micro–nano manufacturing inspection.

SpheriCol: A Driving Assistant for Power Wheelchairs Based on Spherical Vision

This paper presents a new vision-based navigation assistant for power wheelchairs, called SpheriCol. Inspired by the parking assist system in cars, an omnidirectional image captured by a twin-fisheye camera is overlaid with colored distance markers from range sensors mounted on the wheelchair, to generate an augmented view of the surrounding environment. Such an image is displayed in real time to the user on a screen. To evaluate the effectiveness of the proposed system, able-bodied subjects and older adults with motor impairments have been asked to drive a consumer-grade power wheelchair equipped with SpheriCol, via a standard joystick. Our clinical trials with different obstacle courses, consistently indicate that SpheriCol is effective in improving safety and comfort, and in supporting a driver’s decision during challenging but prevalent maneuvers, such as reversing out of an elevator, corridor centering, and turning on the spot.

Omnidirectional 3D-DIC method for determining inner deformation in pipelines

An omnidirectional three-dimensional digital image correlation (DIC) system was developed for full 360-degree inner surface deformation measurements using a pyramid reflector to realize panoramic observation. After the coordinate transformation, the 3D displacement and strain distributions were reconstructed. The basic principles, optical arrangements, and experimental procedures were described in detail. Morphology and rigid-body displacement experiments were conducted to evaluate the efficiency and accuracy of the 3D shape and displacement determination. The experimental results agree well with the actual values and the data obtained by multiple measurements using the traditional 3D-DIC method. Moreover, the proposed method was successfully applied to evaluate the inner deformation distributions of pipes subjected to compression loading. The experimental results further validated the applicability of the proposed method.

Equirectangular Image Data Detection, Segmentation and Classification of Varying Sized Traffic Signs: A Comparison of Deep Learning Methods.

There are known limitations in mobile omnidirectional camera systems with an equirectangular projection in the wild, such as momentum-caused object distortion within images, partial occlusion and the effects of environmental settings. The localization, instance segmentation and classification of traffic signs from image data is of significant importance to applications such as Traffic Sign Detection and Recognition (TSDR) and Advanced Driver Assistance Systems (ADAS). Works show the efficacy of using state-of-the-art deep pixel-wise methods for this task yet rely on the input of classical landscape image data, automatic camera focus and collection in ideal weather settings, which does not accurately represent the application of technologies in the wild. We present a new processing pipeline for extracting objects within omnidirectional images in the wild, with included demonstration in a Traffic Sign Detection and Recognition (TDSR) system. We compare Mask RCNN, Cascade RCNN, and Hybrid Task Cascade (HTC) methods, while testing RsNeXt 101, Swin-S and HRNetV2p backbones, with transfer learning for localization and instance segmentation. The results from our multinomial classification experiment show that using our proposed pipeline, given that a traffic sign is detected, there is above a 95% chance that it is classified correctly between 12 classes despite the limitations mentioned. Our results on the projected images should provide a path to use omnidirectional images with image processing to enable the full surrounding awareness from one image source.

Research on Omnidirectional Stereo Measurement Using Convex Mirrors and Vertical Disparity

We propose an omnidirectional measurement method without blind spots by using a convex mirror, which in principle does not cause chromatic aberration, and by using vertical disparity by installing cameras at the top and bottom of the image. In recent years, there has been significant research in the fields of autonomous cars and robots. In these fields, three-dimensional measurements of the surrounding environment have become indispensable. Depth sensing with cameras is one of the most important sensors for recognizing the surrounding environment. Previous studies have attempted to measure a wide range of areas using fisheye and full spherical panoramic cameras. However, these approaches have limitations such as blind spots and the need for multiple cameras to measure all directions. Therefore, this paper describes a stereo camera system that uses a device capable of taking an omnidirectional image with a single shot, enabling omnidirectional measurement with only two cameras. This achievement was challenging to attain with conventional stereo cameras. The results of experiments confirmed an improvement in accuracy of up to 37.4% compared to previous studies. In addition, the system succeeded in generating depth image that can recognize distances in all directions in a single frame, demonstrating the possibility of omnidirectional measurement with two cameras.

Developing a Model to Express Spatial Relationships on Omnidirectional Images for Indoor Space Representation to Provide Location-Based Services

The unavailability and fragmentation of spatial data are challenges in creating realistic representations of objects and environments in the real world, especially indoors. Among the numerous methods for representing indoor space, the existing research has shown the efficiency and effectiveness of using omnidirectional images. However, they lack information on spatial relationships, so spatial datasets such as the Node-Relation Structure (NRS) must be used to provide location-based services (LBS). This study proposes a method for embedding topological relationships on omnidirectional images, and correspondingly extracting NRS data to enable the expression of these relationships on the images. These relationships include the connectivity of relations among the indoor subunits, and the containment of relations between the spaces and indoor facilities on the image data. This model allows for the construction of an image-based indoor space representation for providing LBS. This paper also demonstrates an approach to utilizing these datasets through an image-based platform that enables the direct performance of spatial analysis relevant to LBS on the images, and provides the accurate visualization and expression of the spaces and indoor points of interest data representing indoor facilities. This paper also includes an experimental implementation to demonstrate the potential of our model for providing an efficient space representation and the handling of basic spatial queries for indoor space applications.

Omni-directional Multi-view Image Measurement System in the Co-sphere Framework

Omni-directional Multi-view Image Measurement System in the Co-sphere Framework

Developing a Method to Detect Indoor Spatial Entities in Omnidirectional Images for Constructing IndoorGML Data

The continuous increase in interest in indoor space demands the development of LBSs (Location Based Services) for navigation, routing, and space query in these environments. Applications providing these services must use visualization and topology data together to provide relevant services to users. Applications have frequently used geometric object-based visualization models for indoor space LBSs. These have the advantage of being easy to use with IndoorGML, an indoor topology model, but have the disadvantage of high construction cost and heavy data. Correspondingly, the image-based visualization model drew attention as an alternative model. However, using such models requires identifying objects on the image to be used alongside IndoorGML, presenting a limitation because it is difficult to identify objects in the image pixels directly. To overcome the limitations of image-based visualization models and reconsider the usability of indoor space LBS using images, this study presents a method to automatically detect spatial objects required to construct IndoorGML from images using deep learning. This methodology aims to detect objects mainly used in indoor LBSs, which include door and stair objects portrayed in indoor omnidirectional images. This study proposes a detailed method of constructing a training dataset for indoor spatial object detection. Moreover, this research presents a method of refining the training dataset and directly acquiring omnidirectional image data to train it as an object detection model that can be used universally in various buildings while maintaining compliant accuracy.