Fisheye Camera Research Articles

Neural Radiance Fields for Fisheye Driving Scenes Using Edge-Aware Integrated Depth Supervision

Neural radiance fields (NeRF) have become an effective method for encoding scenes into neural representations, allowing for the synthesis of photorealistic views of unseen views from given input images. However, the applicability of traditional NeRF is significantly limited by its assumption that images are captured for object-centric scenes with a pinhole camera. Expanding these boundaries, we focus on driving scenarios using a fisheye camera, which offers the advantage of capturing visual information from a wide field of view. To address the challenges due to the unbounded and distorted characteristics of fisheye images, we propose an edge-aware integration loss function. This approach leverages sparse LiDAR projections and dense depth maps estimated from a learning-based depth model. The proposed algorithm assigns larger weights to neighboring points that have depth values similar to the sensor data. Experiments were conducted on the KITTI-360 and JBNU-Depth360 datasets, which are public and real-world datasets of driving scenarios using fisheye cameras. Experimental results demonstrated that the proposed method is effective in synthesizing novel view images, outperforming existing approaches.

Accuracy of fish-eye camera device for aircraft positioning based on laboratory experiment combining computer graphic and 3D Projection

For the management of airborne mobility devices such as airplanes and UAVs, such as noise assessment, it is important to measure the three-dimensional position of the aircraft, and various technologies have been researched and developed. We have developed an Aircraft Positioning Camera (APC) based on an IoT device equipped with a fisheye camera that can mechanically measure aircraft position at a local area with low cost and with high mobility. The performance of this APC is still uncertain as to the limits of the flight altitude and speed of the aircraft that can be measured. Therefore, a detailed study of its performance is needed. Since, it is difficult to standardize the conditions such as weather and ground surface conditions, it is effective to conduct these studies in a laboratory where environmental changes can be controlled. The performance of this APC was examined at an experimental facility where images captured by a 360-degree camera or computer graphics can be projected in three dimensions onto a hemispherical dome.

Real-Time estimation of internal and solar heat gains in buildings using deep learning

This paper presents real-time monitoring of dynamic internal and solar heat gains using programmable low-cost cameras and deep learning techniques. For monitoring changes in occupancy, equipment, lighting, and window status in real time, a convolutional neural network (CNN)-based multi-head classification model was developed and trained with High Dynamic Range (HDR) images, collected using a low-cost fisheye camera in offices. A Python-based region of interest (ROI) generation program was developed to predefine the target areas for monitoring. The classification heads of the model were trained from scratch using a small office image dataset first; then, they were fine-tuned using another HDR image dataset collected in a larger open-plan office with more complex scenes, to evaluate the transferability of the model. The weighted mean classification precision and recall results showed that the developed model could classify the detailed status of each target heat gain (occupants, equipment, lighting, windows) in the predefined areas of the scene with great performance. Finally, to evaluate the impact of real-time monitoring of heat gains on energy demand, the open plan office space used for the experimental dataset collection was modeled using EnergyPlus software using (i) commonly assumed fixed schedules for occupancy, equipment and lighting and (ii) real-time monitored dynamic schedules for internal and solar gain components under the same weather conditions. The results showed that recommended fixed schedules may lead to significant errors in estimated internal and solar gains. The largest discrepancy was noted for occupancy and equipment usage, but other categories also showed both underestimation and overestimation of thermal load components. Implementing reliable and continuous monitoring of dynamic internal and solar heat gains is important for efficient demand-driven HVAC control.

Overhead fisheye cameras for indoor monitoring: challenges and recent progress

Overhead fisheye cameras for indoor monitoring: challenges and recent progress

Utilizing Fisheye Cameras and Mass Spectrometry Imaging to Obtain Probing Coordinates for Histological Samples

Detecting chemical compositions in histological samples using mass spectrometry often requires time-consuming user interaction with a probe to ensure it is positioned as closely as possible to a sampling region. Fisheye cameras can obtain images of samples with a wide field-of-view; however, barrel distortion makes it difficult to analyze them as a flat plane. This research project aimed to develop an application that unwarps live images of histological samples captured with a fisheye camera and calculates the exact image-to-real-world probing coordinates with minor user interaction. Python’s PyQt5 and OpenCV libraries were utilized to create interactive software that removes fisheye distortion, finds image-to-real-world coordinates, and segments images. Connected hardware consisted of a Prusa 3D printer, liquid micro junction-surface sampling probe, fisheye camera, and LED light strips. The printer provided a bed for capturing sample images and control of the camera and probe within 1/100 of a millimetre. The software was outlined to process at least 2 photos of checkerboards for the fisheye undistortion, capture a photo of a sample to be unwarped, and require the user to map 4 points from the image to probe coordinates. Using this approach, real-world coordinates could be found for any image location. The application successfully unwarps live photos and converts image-to-real-world coordinates with sub-millimeter accuracy. Users can take new calibration photos, control lighting, save photos, and check undistortion entirely within the application to ensure all parameters are satisfactory. Coordinates of significant regions are displayed or may be manually selected, making it easier to sample specific areas. The success of unwarping relies on the camera’s focus and height, lighting, and object positioning. Future development plans to improve robustness and coordinate accuracy, and reduce the amount of calibration needed.

A comparison of occupancy-sensing and energy-saving performance: CO2 sensors versus fisheye cameras

A number of occupancy-sensing methods have been proposed to date with the goal of saving HVAC energy in commercial spaces by means of demand-controlled ventilation. Currently, CO2 measurement is the most common approach deployed in practice. However, recently a number of other approaches have been proposed, such as using surveillance cameras, depth sensors, thermal imagers, etc. In this paper, we compare the occupancy-sensing performance of high-accuracy commercial CO2 sensors and overhead fisheye cameras supported by deep-learning algorithms. Our experiments are conducted over 3 days in a large university classroom with highly-dynamic occupancy. First, we assess the impact of parameter selection and sensor placement on CO2-to-occupancy conversion accuracy. Then, for the best-performing setup, we compare this accuracy against that of fisheye cameras. Subsequently, we estimate the potential energy savings offered by both systems. Our results show that overhead fisheye cameras produce on average a 20% lower occupancy-estimation error than CO2 sensors in steady-state periods, and almost 70% lower error in transient periods across 4 performance metrics used in this work. In terms of potential energy savings, fisheye cameras offer on average 6 percentage points of additional savings compared to CO2 sensors. Given their relatively low cost and implementation simplicity, CO2 sensors are an attractive approach to saving HVAC energy, however their occupancy-estimation accuracy is highly sensitive to parameter selection that is challenging in practice. Occupancy sensing based on fisheye cameras is not subject to similar parameterization sensitivity and in addition to outperforming CO2 sensors allows precise localization of occupants in a space thus potentially enabling additional space-management and safety/security services, that CO2 sensors cannot offer.

Implementation of a Small-Sized Mobile Robot with Road Detection, Sign Recognition, and Obstacle Avoidance

In this study, under the limited volume of 18 cm × 18 cm × 21 cm, a small-sized mobile robot is designed and implemented. It consists of a CPU, a GPU, a 2D LiDAR (Light Detection And Ranging), and two fisheye cameras to let the robot have good computing processing and graphics processing capabilities. In addition, three functions of road detection, sign recognition, and obstacle avoidance are implemented on this small-sized robot. For road detection, we divide the captured image into four areas and use Intel NUC to perform road detection calculations. The proposed method can significantly reduce the system load and also has a high processing speed of 25 frames per second (fps). For sign recognition, we use the YOLOv4-tiny model and a data augmentation strategy to significantly improve the computing performance of this model. From the experimental results, it can be seen that the mean Average Precision (mAP) of the used model has increased by 52.14%. For obstacle avoidance, a 2D LiDAR-based method with a distance-based filtering mechanism is proposed. The distance-based filtering mechanism is proposed to filter important data points and assign appropriate weights, which can effectively reduce the computational complexity and improve the robot’s response speed to avoid obstacles. Some results and actual experiments illustrate that the proposed methods for these three functions can be effectively completed in the implemented small-sized robot.

Continuous Leaf Area Index (LAI) Observation in Forests: Validation, Application, and Improvement of LAI-NOS

The leaf area index (LAI) is one of the core parameters reflecting the growth status of vegetation. The continuous long-term observation of the LAI is key when assessing the dynamic changes in the energy exchange of ecosystems and the vegetation’s response indicators to climate change. The errors brought about by non-standard operations in manual LAI measurements hinder the further research utilization of this parameter. The long-term automatic LAI observation network is helpful in reducing errors from manual measurements. To further test the applicability of automatic LAI observation instruments in forest environments, this study carried out comparative validation research of the LAI-NOS (LAI automatic network observation system) at the Wanglang Mountain Ecological Remote Sensing Comprehensive Observation Station, China, comparing it with the results measured by the LAI-2200 Plant Canopy Analyzer (LI-COR, Lincoln, NE, USA), the LAI-probe handheld instrument, and a fisheye lens digital camera (DHP method). Instead of using the original “smoothest window” method, a new method, the “sunrise–sunset” method, is used to extract daily LAI-NOS LAI, and the corresponding confidence level is used to filter the data. The results of the data analysis indicate the following: LAI-NOS has a high data stability. The automatically acquired daily data between two consecutive days has a small deviation and significant correlations. Single-angle/multi-angle LAI measurement results of the LAI-NOS have good correlations with the LAI-2200 (R2 = 0.512/R2 = 0.652), the LAI-probe (R2 = 0.692/R2 = 0.619), and the DHP method (R2 = 0.501/R2 = 0.394). The daily LAI obtained from the improved method, when compared to the original method, both show the same vegetation growth trend. However, the improved method has a smaller dispersion. This study confirms the stability and accuracy of automatic observation instruments in mountainous forests, demonstrating the distinct advantages of automatic measurement instruments in the long-term ground observation of LAIs.

A photogrammetric approach for real‐time visual SLAM applied to an omnidirectional system

AbstractThe problem of sequential estimation of the exterior orientation of imaging sensors and the three‐dimensional environment reconstruction in real time is commonly known as visual simultaneous localisation and mapping (vSLAM). Omnidirectional optical sensors have been increasingly used in vSLAM solutions, mainly for providing a wider view of the scene, allowing the extraction of more features. However, dealing with unmodelled points in the hyperhemispherical field poses challenges, mainly due to the complex lens geometry entailed in the image formation process. To address these challenges, the use of rigorous photogrammetric models that appropriately handle the geometry of fisheye lens cameras can overcome these challenges. Thus, this study presents a real‐time vSLAM approach for omnidirectional systems adapting ORB‐SLAM with a rigorous projection model (equisolid‐angle). The implementation was conducted on the Nvidia Jetson TX2 board, and the approach was evaluated using hyperhemispherical images captured by a dual‐fisheye camera (Ricoh Theta S) embedded into a mobile backpack platform. The trajectory covered a distance of 140 m, with the approach demonstrating accuracy better than 0.12 m at the beginning and achieving metre‐level accuracy at the end of the trajectory. Additionally, we compared the performance of our proposed approach with a generic model for fisheye lens cameras.

A Curb-Detection Network with a Tri-Plane BEV Encoder Module for Autonomous Delivery Vehicles

Curb detection tasks play a crucial role in the perception of the autonomous driving environment for logistics vehicles. With the popularity of multi-modal sensors under the BEV (Bird’s Eye View) paradigm, curb detection tasks are increasingly being integrated into multi-task perception networks, achieving robust detection results. This paper modifies and integrates the tri-plane spatial feature representation method of the EG3D network from the field of 3D reconstruction into a BEV-based multi-modal sensor detection network, including LiDAR, pinhole cameras, and fisheye cameras. The system collects a total of 24,350 frames of data under real road conditions for experimentation, proving the effectiveness of the proposed method.

FishSegSSL: A Semi-Supervised Semantic Segmentation Framework for Fish-Eye Images.

The application of large field-of-view (FoV) cameras equipped with fish-eye lenses brings notable advantages to various real-world computer vision applications, including autonomous driving. While deep learning has proven successful in conventional computer vision applications using regular perspective images, its potential in fish-eye camera contexts remains largely unexplored due to limited datasets for fully supervised learning. Semi-supervised learning comes as a potential solution to manage this challenge. In this study, we explore and benchmark two popular semi-supervised methods from the perspective image domain for fish-eye image segmentation. We further introduce FishSegSSL, a novel fish-eye image segmentation framework featuring three semi-supervised components: pseudo-label filtering, dynamic confidence thresholding, and robust strong augmentation. Evaluation on the WoodScape dataset, collected from vehicle-mounted fish-eye cameras, demonstrates that our proposed method enhances the model's performance by up to 10.49% over fully supervised methods using the same amount of labeled data. Our method also improves the existing image segmentation methods by 2.34%. To the best of our knowledge, this is the first work on semi-supervised semantic segmentation on fish-eye images. Additionally, we conduct a comprehensive ablation study and sensitivity analysis to showcase the efficacy of each proposed method in this research.

Vision transformer models to measure solar irradiance using sky images in temperate climates

Solar Irradiance measurements are critical for a broad range of energy systems, including evaluating performance ratios of photovoltaic systems, as well as forecasting power generation. Using sky images to evaluate solar irradiance, allows for a low-cost, low-maintenance, and easy integration into Internet-of-things network, with minimal data loss. This work demonstrates that a vision transformer-based machine learning model can produce accurate irradiance estimates based on sky-images without any auxiliary data being used. The training data utilizes 17 years of global horizontal, diffuse and direct data, based on a high precision pyranometer and pyrheliometer sun-tracked system; in-conjunction with sky images from a standard lens and a fish-eye camera. The vision transformer-based model learns to attend to relevant features of the sky-images and to produce highly accurate estimates for both global horizontal irradiance (RMSE =52 W/m2) and diffuse irradiance (RMSE = 31 W/m2). This work compares the model’s performance on wide field of view all-sky images as well as images from a standard camera and shows that the vision transformer model works best for all-sky images. For images from a normal camera both vision transformer and convolutional architectures perform similarly with the convolution-based architecture showing an advantage for direct irradiance with an RMSE of 155 W/m2.

Omni-OTPE: Omnidirectional Optimal Real-Time Ground Target Position Estimation System for Moving Lightweight Unmanned Aerial Vehicle.

Ground target detection and positioning systems based on lightweight unmanned aerial vehicles (UAVs) are increasing in value for aerial reconnaissance and surveillance. However, the current method for estimating the target's position is limited by the field of view angle, rendering it challenging to fulfill the demands of a real-time omnidirectional reconnaissance operation. To address this issue, we propose an Omnidirectional Optimal Real-Time Ground Target Position Estimation System (Omni-OTPE) that utilizes a fisheye camera and LiDAR sensors. The object of interest is first identified in the fisheye image, and then, the image-based target position is obtained by solving using the fisheye projection model and the target center extraction algorithm based on the detected edge information. Next, the LiDAR's real-time point cloud data are filtered based on position-direction constraints using the image-based target position information. This step allows for the determination of point cloud clusters that are relevant to the characterization of the target's position information. Finally, the target positions obtained from the two methods are fused using an optimal Kalman fuser to obtain the optimal target position information. In order to evaluate the positioning accuracy, we designed a hardware and software setup, mounted on a lightweight UAV, and tested it in a real scenario. The experimental results validate that our method exhibits significant advantages over traditional methods and achieves a real-time high-performance ground target position estimation function.

Generating a full spherical view by modeling the relation between two fisheye images

Full spherical views provide advantages in many applications that use visual information. Dual back-to-back fisheye cameras are receiving much attention to obtain this type of view. However, obtaining a high-quality full spherical view is very challenging. In this paper, we propose a correction step that models the relation between the pixels of the pair of fisheye images in polar coordinates. This correction is implemented during the mapping from the unit sphere to the fisheye image using the equidistant fisheye projection. The objective is that the projections of the same point in the pair of images have the same position on the unit sphere after the correction. In this way, they will also have the same position on the equirectangular coordinate system. Consequently, the discontinuity between the spherical views for blending is minimized. Throughout the manuscript, we show that the angular polar coordinates of the same scene point in the fisheye images are related by a sine function and the radial distance coordinates by a linear function. Also, we propose employing a polynomial as a geometric transformation between the pair of spherical views during the image alignment since the relationship between the matching points of pairs of spherical views is not linear, especially in the top/bottom regions. Quantitative evaluations demonstrate that using the correction step improves the quality of the full spherical view, i.e. IQ MS-SSIM, up to 7%. Similarly, using a polynomial improves the IQ MS-SSIM up to 6.29% with respect to using an affine matrix.

Parking Lot Occupancy Detection using Deep Learning and Fisheye Camera for AIoT System

The combination of Artificial Intelligence and the Internet of Things (AIoT) has gained significant popularity. Deep neural networks (DNNs) have demonstrated remarkable success in various applications. However, deploying complex AI models on embedded boards can pose challenges due to computational limitations and model complexity. This paper presents an AIoT-based system for smart parking lots using edge devices. Our approach involves developing a detection model and a decision tree for occupancy status classification. Specifically, we utilize YOLOv5 for car license plate (LP) detection by verifying the position of the license plate within the parking space.

Measuring Natural Scenes SFR of Automotive Fisheye Cameras

Measuring Natural Scenes SFR of Automotive Fisheye Cameras

Blending of a novel all sky imager model with persistence and a satellite based model for high-resolution irradiance nowcasting

High shares of variable energy sources such as photovoltaics (PV) make balancing network load and generation increasingly challenging. Electricity grids with high PV-penetration benefit from the consideration of intra-minute and intra-hour variabilities via nowcasts (shortest-term forecasts).In this study we present a novel all-sky imager (ASI) nowcasting system which is benchmarked against an established ASI method, a satellite nowcasting system and persistence. Nowcasts of our novel ASI method, satellite method and persistence are subsequently combined to a hybrid model with improved accuracy.ASI systems exploit sky images from fisheye cameras to analyze sky conditions, predict upcoming cloud situations and derive irradiance nowcasts from these. Our ASI method uses a novel machine-learning (ML) based approach that makes direct use of pixel values and other image features. Measurements from a spatially distributed irradiance measuring network of eight stations within a radius of 10km around the camera position are used to train our ASI and our hybrid model and validate our nowcasting results. In the evaluation we put a special focus on irradiance variability conditions. The high share of stable cloud situations within our dataset (∼68 %) results in a relatively good performance of persistence nowcasts. In higher variability situations persistence is quickly outperformed by all other nowcasting methods.Our ASI method clearly outperforms the reference ASI method at all lead times (LTs). It moreover exhibits a significantly lower root mean square error (RMSE) than the satellite-based method up to LT≤11min ahead and a lower mean absolute error (MAE) throughout the entire interval. The hybrid model brings about an RMSE improvement score of 5-13% over the respective optimal individual method.

Calibration method of fisheye camera for high-precision collimation measurement

Calibration method of fisheye camera for high-precision collimation measurement

Calibration of Fisheye Camera with Co-linear Constraint of the Main Optical Axis and Epipolar Line Orthogonality-Transverse Axis

Calibration of Fisheye Camera with Co-linear Constraint of the Main Optical Axis and Epipolar Line Orthogonality-Transverse Axis

Enhanced Vehicle Movement Counting at Intersections via a Self-Learning Fisheye Camera System

Enhanced Vehicle Movement Counting at Intersections via a Self-Learning Fisheye Camera System