Panoramic Depth Research Articles

A Novel Panorama Depth Estimation Framework for Autonomous Driving Scenarios Based on a Vision Transformer.

An accurate panorama depth estimation result is crucial to risk perception in autonomous driving practice. In this paper, an innovative framework is presented to address the challenges of imperfect observation and projection fusion in panorama depth estimation, enabling the accurate capture of distances from surrounding images in driving scenarios. First, the Patch Filling method is proposed to alleviate the imperfect observation of panoramic depth in autonomous driving scenarios, which constructs a panoramic depth map based on the sparse distance data provided by the 3D point cloud. Then, in order to tackle the distortion challenge faced by outdoor panoramic images, a method for image context learning, ViT-Fuse, is proposed and specifically designed for equirectangular panoramic views. The experimental results show that the proposed ViT-Fuse reduces the estimation error by 9.15% on average in driving scenarios compared with the basic method and exhibits more robust and smoother results on the edge details of the depth estimation maps.

PERF: Panoramic Neural Radiance Field From a Single Panorama.

Neural Radiance Field (NeRF) has achieved substantial progress in novel view synthesis given multi-view images. Recently, some works have attempted to train a NeRF from a single image with 3D priors. They mainly focus on a limited field of view with a few occlusions, which greatly limits their scalability to real-world 360-degree panoramic scenarios with large-size occlusions. In this paper, we present PERF, a 360-degree novel view synthesis framework that trains a panoramic neural radiance field from a single panorama. Notably, PERF allows 3D roaming in a complex scene without expensive and tedious image collection. To achieve this goal, we propose a novel collaborative RGBD inpainting method and a progressive inpainting-and-erasing method to lift up a 360-degree 2D scene to a 3D scene. Specifically, we first predict a panoramic depth map as initialization given a single panorama and reconstruct visible 3D regions with volume rendering. Then we introduce a collaborative RGBD inpainting approach into a NeRF for completing RGB images and depth maps from random views, which is derived from an RGB Stable Diffusion model and a monocular depth estimator. Finally, we introduce an inpainting-and-erasing strategy to avoid inconsistent geometry between a newly-sampled view and reference views. The two components are integrated into the learning of NeRFs in a unified optimization framework and achieve promising results. Extensive experiments on Replica and a new dataset PERF-in-the-wild demonstrate the superiority of our PERF over state-of-the-art methods. Our PERF can be widely used for real-world applications, such as panorama-to-3D, text-to-3D, and 3D scene stylization applications.

AFNet: Asymmetric fusion network for monocular panorama depth estimation

Panoramic depth estimation is a crucial technique in 360°perception, which has gained great popularity in recent years due to its ability to provide complete visual coverage of the scene. Panoramic images are commonly presented in the form of equirectangular projection. However, this projection exhibits significant distortions towards the poles, making it unsuitable to directly apply traditional depth estimation frameworks for perspective imaging. The cubemap projection offers distortion-free images on each face but exhibits obvious discontinuities between adjacent faces. Therefore, according to the respective advantages of these two projection methods, we propose an asymmetric fusion network (AFNet), where the fusion information is only fed back to the equirectangular projection branch during the encoding stage to guide the subsequent encoding. We also design a dual projection feature fusion (DPFF) module in our framework to better combine the strengths of both projections. Furthermore, we introduce an edge information enhancement (EIE) module to enhance the edge contours of the final depth map. Experiments on four representative datasets (including both synthetic and real-world scenarios) show that our proposed AFNet achieves favorable performance compared with existing methods.

Deep panoramic depth prediction and completion for indoor scenes

We introduce a novel end-to-end deep-learning solution for rapidly estimating a dense spherical depth map of an indoor environment. Our input is a single equirectangular image registered with a sparse depth map, as provided by a variety of common capture setups. Depth is inferred by an efficient and lightweight single-branch network, which employs a dynamic gating system to process together dense visual data and sparse geometric data. We exploit the characteristics of typical man-made environments to efficiently compress multi-resolution features and find short- and long-range relations among scene parts. Furthermore, we introduce a new augmentation strategy to make the model robust to different types of sparsity, including those generated by various structured light sensors and LiDAR setups. The experimental results demonstrate that our method provides interactive performance and outperforms state-of-the-art solutions in computational efficiency, adaptivity to variable depth sparsity patterns, and prediction accuracy for challenging indoor data, even when trained solely on synthetic data without any fine tuning.

GLPanoDepth: Global-to-Local Panoramic Depth Estimation.

Depth estimation is a fundamental task in many vision applications. With the popularity of omnidirectional cameras, it becomes a new trend to tackle this problem in the spherical space. In this paper, we propose a learning-based method for predicting dense depth values of a scene from a monocular omnidirectional image. An omnidirectional image has a full field-of-view, providing much more complete descriptions of the scene than perspective images. However, fully-convolutional networks that most current solutions rely on fail to capture rich global contexts from the panorama. To address this issue and also the distortion of equirectangular projection in the panorama, we propose Cubemap Vision Transformers (CViT), a new transformer-based architecture that can model long-range dependencies and extract distortion-free global features from the panorama. We show that cubemap vision transformers have a global receptive field at every stage and can provide globally coherent predictions for spherical signals. As a general architecture, it removes any restriction that has been imposed on the panorama in many other monocular panoramic depth estimation methods. To preserve important local features, we further design a convolution-based branch in our pipeline (dubbed GLPanoDepth) and fuse global features from cubemap vision transformers at multiple scales. This global-to-local strategy allows us to fully exploit useful global and local features in the panorama, achieving state-of-the-art performance in panoramic depth estimation.

Effective fusion module with dilation convolution for monocular panoramic depth estimate

AbstractDepth estimation from monocular panoramic image is a crucial step in 3D reconstruction, which is a close relationship with virtual reality and metaverse technologies. In recent years, some methods, such as HRDFuse, BiFuse++, and UniFuse, have employed a two‐branch neural network leveraging two common projections: equirectangular and cubemap projections (CMPs). The equirectangular projection (ERP) provides a complete field of view but introduces distortion, while the CMP avoids distortion but introduces discontinuity at the boundary of the cube. In order to address the issue of distortion and discontinuity, the authors propose an efficient depth estimation fusion module to balance the feature mapping of the two projections. Moreover, for the ERP, the authors propose a novel inflated network architecture to extend the receptive field and effectively harness visual information. Extensive experiments show that the authors’ method predicts more clear boundaries and accurate depth results while outperforming mainstream panoramic depth estimation algorithms.

E2LNet: An Efficient and Effective Lightweight Network for Panoramic Depth Estimation.

Monocular panoramic depth estimation has various applications in robotics and autonomous driving due to its ability to perceive the entire field of view. However, panoramic depth estimation faces two significant challenges: global context capturing and distortion awareness. In this paper, we propose a new framework for panoramic depth estimation that can simultaneously address panoramic distortion and extract global context information, thereby improving the performance of panoramic depth estimation. Specifically, we introduce an attention mechanism into the multi-scale dilated convolution and adaptively adjust the receptive field size between different spatial positions, designing the adaptive attention dilated convolution module, which effectively perceives distortion. At the same time, we design the global scene understanding module to integrate global context information into the feature maps generated using the feature extractor. Finally, we trained and evaluated our model on three benchmark datasets which contains the virtual and real-world RGB-D panorama datasets. The experimental results show that the proposed method achieves competitive performance, comparable to existing techniques in both quantitative and qualitative evaluations. Furthermore, our method has fewer parameters and more flexibility, making it a scalable solution in mobile AR.

SPDET: Edge-Aware Self-Supervised Panoramic Depth Estimation Transformer With Spherical Geometry.

Panoramic depth estimation has become a hot topic in 3D reconstruction techniques with its omnidirectional spatial field of view. However, panoramic RGB-D datasets are difficult to obtain due to the lack of panoramic RGB-D cameras, thus limiting the practicality of supervised panoramic depth estimation. Self-supervised learning based on RGB stereo image pairs has the potential to overcome this limitation due to its low dependence on datasets. In this work, we propose the SPDET, an edge-aware self-supervised panoramic depth estimation network that combines the transformer with a spherical geometry feature. Specifically, we first introduce the panoramic geometry feature to construct our panoramic transformer and reconstruct high-quality depth maps. Furthermore, we introduce the pre-filtered depth-image-based rendering method to synthesize the novel view image for self-supervision. Meanwhile, we design an edge-aware loss function to improve the self-supervised depth estimation for panorama images. Finally, we demonstrate the effectiveness of our SPDET with a series of comparison and ablation experiments while achieving the state-of-the-art self-supervised monocular panoramic depth estimation. Our code and models are available at https://github.com/zcq15/SPDET.

Monocular catadioptric panoramic depth estimation via improved end-to-end neural network model.

In this paper, we propose a monocular catadioptric panoramic depth estimation algorithm based on an improved end-to-end neural network model. First, we use an enhanced concentric circle approximation unfolding algorithm to unfold the panoramic images captured by the catadioptric panoramic camera and then extract the effective regions. In addition, the integration of the Non-local attention mechanism is exploited to improve image understanding. Finally, a depth smoothness loss strategy is implemented to further enhance the reliability and precision of the estimated depths. Experimental results confirm that this refined algorithm is capable of providing highly accurate estimates of object depth.

Confocal microscopy multi-focus image fusion method based on axial information guidance.

Aiming at the problems of poor anti-interference of existing pixel-level fusion rules and low efficiency of transform domain fusion rules, this study proposes a confocal microscopic multi-focus image fusion method (IGCM) based on differential confocal axial information guidance. Unlike traditional multi-focus image fusion (MFIF) methods, IGCM uses height information rather than grayscale or frequency to determine clear areas. First, the differential confocal axial measurement curve is calibrated to determine the suitable scan step u. Second, the image set required for fusion is constructed by performing a hierarchical scan of the measurement samples. Then, multiple differential image pairs are constructed using the step size u and the set of images, and the extraction area of the current reference image is decided based on the height obtained from the differential image. Finally, the regions determined by each reference image are extracted and the duplicated pixels are averaged to obtain the MFIF image. The results were that IGCM improves the interference immunity based on pixel-level image fusion compared to the maximum peak fusion method. Compared with other MFIFs, IGCM has excellent fusion efficiency while ensuring fusion clarity, which can meet the application scenario of real-time fusion and offers a new approach to panoramic depth images for confocal devices.

Review on Panoramic Imaging and Its Applications in Scene Understanding

With the rapid development of high-speed communication and artificial intelligence technologies, human perception of real-world scenes is no longer limited to the use of small Field of View (FoV) and low-dimensional scene detection devices. Panoramic imaging emerges as the next generation of innovative intelligent instruments for environmental perception and measurement. However, while satisfying the need for large-FoV photographic imaging, panoramic imaging instruments are expected to have high resolution, no blind area, miniaturization, and multidimensional intelligent perception, and can be combined with artificial intelligence methods towards the next generation of intelligent instruments, enabling deeper understanding and more holistic perception of 360° real-world surrounding environments. Fortunately, recent advances in freeform surfaces, thin-plate optics, and metasurfaces provide innovative approaches to address human perception of the environment, offering promising ideas beyond conventional optical imaging. In this review, we begin with introducing the basic principles of panoramic imaging systems, and then describe the architectures, features, and functions of various panoramic imaging systems. Afterwards, we discuss in detail the broad application prospects and great design potential of freeform surfaces, thin-plate optics, and metasurfaces in panoramic imaging. We then provide a detailed analysis on how these techniques can help enhance the performance of panoramic imaging systems. We further offer a detailed analysis of applications of panoramic imaging in scene understanding for autonomous driving and robotics, spanning panoramic semantic image segmentation, panoramic depth estimation, panoramic visual localization, and so on. Finally, we cast a perspective on future potential and research directions for panoramic imaging instruments.

Research on DSO vision positioning technology based on binocular stereo panoramic vision system

In the visual positioning of Unmanned Ground Vehicle (UGV), the visual odometer based on direct sparse method (DSO) has the advantages of small amount of calculation, high real-time performance and high robustness, so it is more widely used than the visual odometer based on feature point method. Ordinary vision sensors have a narrower viewing angle than panoramic vision sensors, and there are fewer road signs in a single frame of image, resulting in poor road sign tracking and positioning capabilities, and severely restricting the development of visual odometry. Based on these considerations, this paper proposes a binocular stereo panoramic vision positioning algorithm based on extended DSO, which can solve these problems well. The experimental results show that the binocular stereo panoramic vision positioning algorithm based on the extended DSO can directly obtain the panoramic depth image around the UGV, which greatly improves the accuracy and robustness of the visual positioning compared with other ordinary visual odometers. It will have widely application prospects in the UGV field in the future.

Panoramic depth estimation via supervised and unsupervised learning in indoor scenes.

Depth estimation, as a necessary clue to convert 2D images into the 3D space, has been applied in many machine vision areas. However, to achieve an entire surrounding 360° geometric sensing, traditional stereo matching algorithms for depth estimation are limited due to large noise, low accuracy, and strict requirements for multi-camera calibration. In this work, for a unified surrounding perception, we introduce panoramic images to obtain a larger field of view. We extend PADENet [IEEE 23rd International Conference on Intelligent Transportation Systems, (2020), pp. 1-610.1109/ITSC45102.2020.9294206], which first appeared in our previous conference work for outdoor scene understanding, to perform panoramic monocular depth estimation with a focus for indoor scenes. At the same time, we improve the training process of the neural network adapted to the characteristics of panoramic images. In addition, we fuse the traditional stereo matching algorithm with deep learning methods and further improve the accuracy of depth predictions. With a comprehensive variety of experiments, this research demonstrates the effectiveness of our schemes aiming for indoor scene perception.

Unsupervised Learning of Depth and Ego-Motion from Cylindrical Panoramic Video with Applications for Virtual Reality

We introduce a convolutional neural network model for unsupervised learning of depth and ego-motion from cylindrical panoramic video. Panoramic depth estimation is an important technology for applications such as virtual reality, 3D modeling, and autonomous robotic navigation. In contrast to previous approaches for applying convolutional neural networks to panoramic imagery, we use the cylindrical panoramic projection which allows for the use of the traditional CNN layers such as convolutional filters and max pooling without modification. Our evaluation of synthetic and real data shows that unsupervised learning of depth and ego-motion on cylindrical panoramic images can produce high-quality depth maps and that an increased field-of-view improves ego-motion estimation accuracy. We create two new datasets to evaluate our approach: a synthetic dataset created using the CARLA simulator, and Headcam, a novel dataset of panoramic video collected from a helmet-mounted camera while biking in an urban setting. We also apply our network to the problem of converting monocular panoramas to stereo panoramas.

Monitoring Technology of Power System Operation State Based on Stereo Sensing

In recent years, the development of computer technology, automation control technology and communication technology has provided new hardware and software support for the implementation of distribution network operation monitoring. In the daily production of the oil field, the reliability and stability of the power supply are the key links to ensure the completion of the production tasks of the oil field. This paper studies the problem of scene depth reconstruction in panoramic technology. On this basis, the panoramic depth map is combined with the panoramic image, and the three-dimensional panoramic characteristics are reconstructed from the visual and environmental aspects of the stereo perception model suitable for its performance. This paper develops an oilfield distribution network operation monitoring system based on stereo perception, and develops corresponding system management software. This paper first determines the mathematical model of the distribution network line loss calculation, and introduces the principle of GPRS technology to formulate a specific network plan for distribution network operation monitoring. Finally, based on the terminal data collection method of CDD-IDT / GPRS, the development of terminal system management software is finally completed. Field test results show that the stereoscopic perception-based oil field distribution network operation monitoring system developed in this paper has high accuracy of accident prediction, strong real-time performance, stable and reliable operation, and the average transmission time of each data in the stereoscopic perception based monitoring network transmission It is 1.88 seconds, which can be displayed immediately after pre-loading.

Pedestrian Detection Based on Panoramic Depth Map Transformed from 3D-LiDAR Data

Object detection is a crucial task of autonomous driving. This paper addresses an effective algorithm for pedestrian detection of the panoramic depth map transformed from the 3D-LiDAR data. Firstly, the 3D point clouds are transformed into panoramic depth maps, and then the panoramic depth maps are enhanced. Secondly, the grounds of the 3D point clouds are removed. The remaining point clouds are clustered, filtered and projected onto the previously generated panoramic depth maps, and new panoramic depth maps are obtained. Finally, the new panoramic depth maps are jointed to generate depth maps with different sizes, which are used as input of the improved PVANET for pedestrian detection. The 2D image of the panoramic depth map applied to the proposed algorithm is transformed from 3D point cloud, effectively containing the panorama of the sensor, and is more suitable for the environment perception of autonomous driving. Compared with the detection algorithm based on RGB images, the proposed algorithm cannot be affected by light, and can maintain the normal average precision of pedestrian detection at night. In order to increase the robustness of detecting small objects like pedestrians, the network structure based on the original PVANET is modified in this paper. A new dataset is built by processing the 3D-LiDAR data and the model trained on the new dataset perform well. The experimental results show that the proposed algorithm achieves high accuracy and robustness in pedestrian detection under different illumination conditions. Furthermore, when trained on the new dataset, the model exhibits average precision improvements of 2.8–5.1 % over the original PVANET, making it more suitable for autonomous driving applications.

Research and Application of 3D Face Modeling Algorithm Based on ICP Accurate Alignment

This paper focuses on 3D face modeling. First, the panorama is divided into faces. Then the segmented face depth map is filtered first and then registered. Finally, complete the process of modeling the human face. The face segmentation process is mainly to remove the background of the panoramic depth map, and then perform segmentation of the human head. In this paper, select the rough registration method based on SIFT feature point depth map.Use the optimized bidirectional matching algorithm to purify the matched feature point pairs. Then this paper uses the SVD algorithm to find the transformation matrix and complete the depth map rough registration. This paper mainly selects the improved ICP algorithm for accurate registration. The improvement strategy is mainly based on the search acceleration of the K-D tree. The experimental results show that the curvature of the human body changes greatly between the frontal and lateral surfaces of the human body. When the overlapping area is small, the face registration can be performed more accurately and complete face modeling can be performed.

Landmark-Based Homing Navigation Using Omnidirectional Depth Information.

A number of landmark-based navigation algorithms have been studied using feature extraction over the visual information. In this paper, we apply the distance information of the surrounding environment in a landmark navigation model. We mount a depth sensor on a mobile robot, in order to obtain omnidirectional distance information. The surrounding environment is represented as a circular form of landmark vectors, which forms a snapshot. The depth snapshots at the current position and the target position are compared to determine the homing direction, inspired by the snapshot model. Here, we suggest a holistic view of panoramic depth information for homing navigation where each sample point is taken as a landmark. The results are shown in a vector map of homing vectors. The performance of the suggested method is evaluated based on the angular errors and the homing success rate. Omnidirectional depth information about the surrounding environment can be a promising source of landmark homing navigation. We demonstrate the results that a holistic approach with omnidirectional depth information shows effective homing navigation.

Monocular catadioptric panoramic depth estimation via caustics-based virtual scene transition.

Existing catadioptric panoramic depth estimation systems usually require two panoramic imaging subsystems to achieve binocular disparity. The system structures are complicated and only sparse depth maps can be obtained. We present a novel monocular catadioptric panoramic depth estimation method that achieves dense depth maps of panoramic scenes using a single unmodified conventional catadioptric panoramic imaging system. Caustics model the reflection of the curved mirror and establish the distance relationship between the virtual and real panoramic scenes to overcome the nonlinear problem of the curved mirror. Virtual scene depth is then obtained by applying our structure classification regularization to depth from defocus. Finally, real panoramic scene depth is recovered using the distance relationship. Our method's effectiveness is demonstrated in experiments.

Directions for Public Sociology: Novel Writing as a Creative Approach

This article presents a creative direction for public sociology: novel writing. Narrativity is embedded within much contemporary sociological work, and sociologists and novelists share a number of complementary approaches for understanding and interpreting the social world. This article argues that novel writing presents sociologists with a process and medium through which they can expand their work for a more public, engaging, affective, and panoramic sociology. Here, the historical development of sociological thought is considered as well as the recent progress of public sociology. Three key strengths of sociological novels are presented: promoting public sociology and interlocutor engagement; transforming knowledge exchange from mimetic to sympractic communication; and addressing issues of scope. Two recent sociological novels are discussed: Blue by Patricia Leavy and On The Cusp by David Buckingham, both published in 2015. Finally, two linked aspects for (thinking about) writing sociological fiction are explored: the concept of glocality and the methodology of ethnography. Employing creative mediums such as novels as public sociology may cultivate a wider, affective public engagement with significant academic ideas such as the sociological imagination. Sociological novels work to bring the local and global into dialogue, and may help achieve the scope and panoramic depth that sociology requires.