Fisheye Images Research Articles

Neural Radiance Fields for Fisheye Driving Scenes Using Edge-Aware Integrated Depth Supervision.

Neural radiance fields (NeRF) have become an effective method for encoding scenes into neural representations, allowing for the synthesis of photorealistic views of unseen views from given input images. However, the applicability of traditional NeRF is significantly limited by its assumption that images are captured for object-centric scenes with a pinhole camera. Expanding these boundaries, we focus on driving scenarios using a fisheye camera, which offers the advantage of capturing visual information from a wide field of view. To address the challenges due to the unbounded and distorted characteristics of fisheye images, we propose an edge-aware integration loss function. This approach leverages sparse LiDAR projections and dense depth maps estimated from a learning-based depth model. The proposed algorithm assigns larger weights to neighboring points that have depth values similar to the sensor data. Experiments were conducted on the KITTI-360 and JBNU-Depth360 datasets, which are public and real-world datasets of driving scenarios using fisheye cameras. Experimental results demonstrated that the proposed method is effective in synthesizing novel view images, outperforming existing approaches.

High-resolution in vivo 4D-OCT fish-eye imaging using 3D-UNet with multi-level residue decoder.

Optical coherence tomography (OCT) allows high-resolution volumetric imaging of biological tissues in vivo. However, 3D-image acquisition often suffers from motion artifacts due to slow frame rates and involuntary and physiological movements of living tissue. To solve these issues, we implement a real-time 4D-OCT system capable of reconstructing near-distortion-free volumetric images based on a deep learning-based reconstruction algorithm. The system initially collects undersampled volumetric images at a high speed and then upsamples the images in real-time by a convolutional neural network (CNN) that generates high-frequency features using a deep learning algorithm. We compare and analyze both dual-2D- and 3D-UNet-based networks for the OCT 3D high-resolution image reconstruction. We refine the network architecture by incorporating multi-level information to accelerate convergence and improve accuracy. The network is optimized by utilizing the 16-bit floating-point precision for network parameters to conserve GPU memory and enhance efficiency. The result shows that the refined and optimized 3D-network is capable of retrieving the tissue structure more precisely and enable real-time 4D-OCT imaging at a rate greater than 10 Hz with a root mean square error (RMSE) of ∼0.03.

Swin‐fisheye: Object detection for fisheye images

AbstractFisheye cameras have been widely used in autonomous navigation, visual surveillance, and automatic driving. Due to severe geometric distortion, fisheye images cannot be processed effectively by conventional methods. The existing object detection algorithms cannot better detect the small targets or the objects with large distortion in the fisheye images. The size and scene of available fisheye datasets (such as WoodScape and VOC‐360) cannot satisfy the training of robust network models. Herein, the authors propose Swin‐Fisheye, an end‐to‐end object detection algorithm based on Swin Transformer. A feature pyramid module based on deformable convolution (DFPM) is designed to obtain richer contextual information from the multi‐scale feature maps. In addition, a projection transformation algorithm (PTA) is proposed, which can convert rectilinear images into fisheye images more accurately, and then create a fisheye image dataset (COCO‐Fish). The results of extensive experiments conducted on VOC‐360, WoodScape, and COCO‐Fish demonstrate that the proposed algorithm can achieve satisfactory results compared with state‐of‐the‐art methods.

Fisheye Night Sky Imager: A Calibrated Tool to Measure Night Sky Brightness

The Night Skies Team of the U.S. National Park Service (NPS) has developed a new camera system equipped with a fisheye lens to measure night sky brightness. This NPS Fisheye Night Sky Imager comprises a back-illuminated Sony IMX455 CMOS sensor housed in the ZWO ASI6200MM camera, a Johnson V filter, and a Sigma 8 mm F3.5 fisheye lens. All of the components are commercially available. The fisheye lens allows us to capture the entire sky in a single frame. The wide-angle view also introduces calibration challenges. To address this, we have customized tools and devised innovative methods for calibrating the system, including flat-field correction, pixel scale determination, and field of view characterization. Additionally, we use standard stars for precise positional and photometric calibration. Furthermore, we create an open-source Python pipeline to process these fisheye images and leverage Git and GitHub for source code version control and distribution. The resulting images provide a positionally and brightness-calibrated fisheye view of the night sky, with a photometric calibration uncertainty of 0.12 mag. When comparing our Fisheye Night Sky Imager measurements to images taken with other calibrated systems, we reliably achieve consistent results under both dark and bright skies. Our Fisheye Night Sky Imager not only enables rapid assessment of sky quality but also supports the feasibility of establishing permanent monitoring locations. Recognizing that the natural dark sky is a critical element for natural, cultural, educational, and visitor experiences, this new camera system enhances our ability to assess resource conditions and fulfill the NPS mission of conserving resources unimpaired for the enjoyment of this and future generations.

A correction method for calculating sky view factor in urban canyons using fisheye images

The Sky View Factor (SVF) is a critical parameter for studying the thermal environment of buildings theoretically. However, there are two main definitions of SVF, one based on the visible area ratio and the other based on the view factor. When utilizing fisheye images for SVF calculation, various issues often arise, such as conceptual confusion and algorithmic errors, significantly impeding the accurate scientific analysis of thermal environmental problems. Therefore, in this study, we employed methods such as theoretical model deduction, field analysis, and software simulation and validation to investigate the calculation methods for fisheye images with equidistant projection, equisolid angle projection, and stereographic projection. We proposed the Image Distance Ratio Transformation Method (IDRTM), a correction method for calculating the SVF of fisheye images with the three projection types and corresponding calculation formulas. Finally, two cases demonstrate the computational advantages of simplicity and high accuracy associated with the orthogonal transformation formulas applied to the three types of fisheye images. In this study, the error rates of SVF calculation using three non-orthographic fisheye images directly are all higher than 29 %. After applying IDRTM, the average error rates of the transformed IDRs of three types of fisheye images are all less than 0.58 %, and the error rates of the corrected SVF of three types of fisheye images are all less than 1 %. This study elucidates the scientific definition and algorithm of SVF and provides a scientific basis for the accurate selection of SVF calculation methods corresponding to fisheye image types.

DaFIR: Distortion-Aware Representation Learning for Fisheye Image Rectification

DaFIR: Distortion-Aware Representation Learning for Fisheye Image Rectification

OmniGlasses: an optical aid for stereo vision CNNs to enable omnidirectional image processing

Stereo vision is a key technology for 3D scene reconstruction from image pairs. Most approaches process perspective images from commodity cameras. These images, however, have a very limited field of view and only picture a small portion of the scene. In contrast, omnidirectional images, also known as fisheye images, exhibit a much larger field of view and allow a full 3D scene reconstruction with a small amount of cameras if placed carefully. However, omnidirectional images are strongly distorted which make the 3D reconstruction much more sophisticated. Nowadays, a lot of research is conducted on CNNs for omnidirectional stereo vision. Nevertheless, a significant gap between estimation accuracy and throughput can be observed in the literature. This work aims to bridge this gap by introducing a novel set of transformations, namely OmniGlasses. These are incorporated into the architecture of a fast network, i.e., AnyNet, originally designed for scene reconstruction on perspective images. Our network, Omni-AnyNet, produces accurate omnidirectional distance maps with a mean absolute error of around 13 cm at 48.4 fps and is therefore real-time capable.

Analysis of the Photogrammetric Use of 360-Degree Cameras in Complex Heritage-Related Scenes: Case of the Necropolis of Qubbet el-Hawa (Aswan Egypt).

This study shows the results of the analysis of the photogrammetric use of 360-degree cameras in complex heritage-related scenes. The goal is to take advantage of the large field of view provided by these sensors and reduce the number of images used to cover the entire scene compared to those needed using conventional cameras. We also try to minimize problems derived from camera geometry and lens characteristics. In this regard, we used a multi-sensor camera composed of six fisheye lenses, applying photogrammetric procedures to several funerary structures. The methodology includes the analysis of several types of spherical images obtained using different stitching techniques and the comparison of the results of image orientation processes considering these images and the original fisheye images. Subsequently, we analyze the possible use of the fisheye images to model complex scenes by reducing the use of ground control points, thus minimizing the need to apply surveying techniques to determine their coordinates. In this regard, we applied distance constraints based on a previous extrinsic calibration of the camera, obtaining results similar to those obtained using a traditional schema based on points. The results have allowed us to determine the advantages and disadvantages of each type of image and configuration, providing several recommendations regarding their use in complex scenes.

Developing an expansion‐based obstacle detection using panoptic segmentation

AbstractSafe Micro Aerial Vehicle (MAV) navigation requires detecting and avoiding obstacles. For safe MAV navigation, expansion‐based algorithms are effective for detecting obstacles. However, accurate and real‐time obstacle detection is a fundamental challenge. Some traditional methods focus on extracting geometric features from images and applying geometric constraints to identify potential obstacles. Others may leverage machine learning algorithms for object detection and classification, using features such as texture, shape, and context to distinguish obstacles from background clutter. The choice of approach depends on factors such as the specific requirements of the application, the complexity of the scene, and the available computational resources. Since obstacles, in reality, take the form of objects (e.g., persons, walls, pillars, trees, automobiles, and other structures), it is preferable to represent them according to human comprehension and as objects. Therefore, the objective of this study is to reflect on the previous research and address the issues mentioned above by extracting objects from the fisheye image using a panoptic deep‐learning network. The extracted object regions are, then, used to identify obstacles with a novel area‐based expansion rate we developed in a previous study. We compared the accuracy of obstacle detection in our proposed method to the existing method when moving forward and to the right; thus, we improved it between 10% and 18%, respectively. In addition, compared with the existing method, and due to replacing a single object with multiple regions, obstacle‐detection runtime for forward and right direction is 15.71 and 25.5 times faster, respectively, and the required match points have decreased by 49% and 55%.

FishSegSSL: A Semi-Supervised Semantic Segmentation Framework for Fish-Eye Images.

The application of large field-of-view (FoV) cameras equipped with fish-eye lenses brings notable advantages to various real-world computer vision applications, including autonomous driving. While deep learning has proven successful in conventional computer vision applications using regular perspective images, its potential in fish-eye camera contexts remains largely unexplored due to limited datasets for fully supervised learning. Semi-supervised learning comes as a potential solution to manage this challenge. In this study, we explore and benchmark two popular semi-supervised methods from the perspective image domain for fish-eye image segmentation. We further introduce FishSegSSL, a novel fish-eye image segmentation framework featuring three semi-supervised components: pseudo-label filtering, dynamic confidence thresholding, and robust strong augmentation. Evaluation on the WoodScape dataset, collected from vehicle-mounted fish-eye cameras, demonstrates that our proposed method enhances the model's performance by up to 10.49% over fully supervised methods using the same amount of labeled data. Our method also improves the existing image segmentation methods by 2.34%. To the best of our knowledge, this is the first work on semi-supervised semantic segmentation on fish-eye images. Additionally, we conduct a comprehensive ablation study and sensitivity analysis to showcase the efficacy of each proposed method in this research.

OARPD: occlusion-aware rotated people detection in overhead fisheye images

OARPD: occlusion-aware rotated people detection in overhead fisheye images

Omni-OTPE: Omnidirectional Optimal Real-Time Ground Target Position Estimation System for Moving Lightweight Unmanned Aerial Vehicle.

Ground target detection and positioning systems based on lightweight unmanned aerial vehicles (UAVs) are increasing in value for aerial reconnaissance and surveillance. However, the current method for estimating the target's position is limited by the field of view angle, rendering it challenging to fulfill the demands of a real-time omnidirectional reconnaissance operation. To address this issue, we propose an Omnidirectional Optimal Real-Time Ground Target Position Estimation System (Omni-OTPE) that utilizes a fisheye camera and LiDAR sensors. The object of interest is first identified in the fisheye image, and then, the image-based target position is obtained by solving using the fisheye projection model and the target center extraction algorithm based on the detected edge information. Next, the LiDAR's real-time point cloud data are filtered based on position-direction constraints using the image-based target position information. This step allows for the determination of point cloud clusters that are relevant to the characterization of the target's position information. Finally, the target positions obtained from the two methods are fused using an optimal Kalman fuser to obtain the optimal target position information. In order to evaluate the positioning accuracy, we designed a hardware and software setup, mounted on a lightweight UAV, and tested it in a real scenario. The experimental results validate that our method exhibits significant advantages over traditional methods and achieves a real-time high-performance ground target position estimation function.

Generating a full spherical view by modeling the relation between two fisheye images

Full spherical views provide advantages in many applications that use visual information. Dual back-to-back fisheye cameras are receiving much attention to obtain this type of view. However, obtaining a high-quality full spherical view is very challenging. In this paper, we propose a correction step that models the relation between the pixels of the pair of fisheye images in polar coordinates. This correction is implemented during the mapping from the unit sphere to the fisheye image using the equidistant fisheye projection. The objective is that the projections of the same point in the pair of images have the same position on the unit sphere after the correction. In this way, they will also have the same position on the equirectangular coordinate system. Consequently, the discontinuity between the spherical views for blending is minimized. Throughout the manuscript, we show that the angular polar coordinates of the same scene point in the fisheye images are related by a sine function and the radial distance coordinates by a linear function. Also, we propose employing a polynomial as a geometric transformation between the pair of spherical views during the image alignment since the relationship between the matching points of pairs of spherical views is not linear, especially in the top/bottom regions. Quantitative evaluations demonstrate that using the correction step improves the quality of the full spherical view, i.e. IQ MS-SSIM, up to 7%. Similarly, using a polynomial improves the IQ MS-SSIM up to 6.29% with respect to using an affine matrix.

Achieving widely distributed feature matches using flattened-affine-SIFT algorithm for fisheye images.

Performing correction first is the most common methods to address feature matching issues for fisheye images, but corrections often result in significant loss of scene details or stretching of images, leaving peripheral regions without matches. In this paper, we propose a novel approach, named flattened-affine-SIFT, to find widely distributed feature matches between stereo fisheye images. Firstly, we establish a new imaging model that integrates a scalable model and a hemisphere model. Utilizing the extensibility of the imaging model, we design a flattened array model to reduce the distortion of fisheye images. Additionally, the affine transformation is performed on the flattened simulation images, which are computed using the differential expansion and the optimal rigidity transformation. Then feature matches are extracted and matched from the simulated images. Experiments on indoor and outdoor fisheye images show that the proposed algorithm can find a large number of reliable feature matches. Moreover, these matches tend to be dispersed over the entire effective image, including peripheral regions with dramatic distortion.

An adversarial pedestrian detection model based on virtual fisheye image training

An adversarial pedestrian detection model based on virtual fisheye image training

Key point calibrating and clustering for hard example mining of dense analogs in the fish-eye lens.

Recent years have witnessed widespread applications of the fish-eye lens with a wide field-of-view. However, its inherent distortion poses a big challenge to the intelligent recognition of dense analogs (IRDA) by convolutional neural networks (CNN). The major bottleneck of existing CNN models lies in their limited modeling capacity for distorted objects in fish-eye images, leading to the misclassification of hard examples. To further improve the accuracy of IRDA, we propose a novel key point calibrating and clustering (KPCC) algorithm based on the hemispherical projection model. Our method can effectively correct the hard example misclassification predicted by the CNN, significantly enhancing the performance of the IRDA. The experiments show that, as a light-weight computation calibrating and stable adaptive clustering method, the KPCC increases the precision and recall rate of IRDA on the intelligent retail dataset by 8.55% and 8.07%, respectively; compared with the classic Focalloss, QFocalloss, and OHEM (online hard example mining), it can mine hard examples more sufficiently, especially in the scene of distorted dense analog detection.

PENERAPAN METODE NAÏVE BAYES DALAM KLASIFIKASI KESEGARAN IKAN BERDASARKAN WARNA PADA CITRA AREA MATA

As a maritime nation, fish is a staple in the Indonesian diet, rich in nutrition and a crucial protein source. It is imperative to maintain the freshness of fish to ensure the quality of fish production. However, the practice of mixing fresh and non-fresh fish poses a serious threat to consumer health and diminishes the overall quality of fish production. Therefore, the development of an automated and efficient method is necessary to distinguish between fresh and non-fresh fish. This research proposes the application of the Naïve Bayes method in classifying fish freshness based on color analysis in the eye area image. This approach involves the extraction of entropy features after segmenting fish images using the RGB and YCbCr color models. A total of 40 datasets of fish eye images were used for training and testing the model. The research results indicate that the proposed classification method achieved an accuracy rate of 97.5%. This success signifies the potential of the color analysis method and entropy features in distinguishing levels of fish freshness. These findings contribute to the development of automated techniques for monitoring and processing fish quality in the fisheries industry.

Extraction of Feature Point Pseudo-Neighborhood for High-Accuracy of Fisheye Image Matching

Extraction of Feature Point Pseudo-Neighborhood for High-Accuracy of Fisheye Image Matching

Improving Passenger Detection With Overhead Fisheye Imaging

Improving Passenger Detection With Overhead Fisheye Imaging

Research on Semantic Segmentation of Fish-Eye Images for Autonomous Driving

Abstract: Fisheye cameras, valued for their wide field of view, play a crucial role in perceiving the surrounding environment of vehicles. However, there is a lack of specific research addressing the processing of significant distortion features in segmenting fish-eye images. Additionally, fish-eye images for autonomous driving face the challenge of few datasets, potentially causing over fitting and hindering the model's generalization ability.
 Based on the semantic segmentation task, a method for transforming normal images into fish-eye images is proposed, which expands the fish-eye image dataset. By employing the Transformer network and the Across Feature Map Attention, the segmentation performance is further improved, achieving a 55.6% mIOU on Woodscape. Additionally, leveraging the concept of knowledge distillation, the network ensures a strong generalization based on dual-domain learning without compromising performance on Woodscape (54% mIOU).