Fisheye Images Research Articles

Fisheye Image Detection of Trees Using Improved YOLOX for Tree Height Estimation.

Tree height is an essential indicator in forestry research. This indicator is difficult to measure directly, as well as wind disturbance adds to the measurement difficulty. Therefore, tree height measurement has always been an issue that experts and scholars strive to improve. We propose a tree height measurement method based on tree fisheye images to improve the accuracy of tree height measurements. Our aim is to extract tree height extreme points in fisheye images by proposing an improved lightweight target detection network YOLOX-tiny. We added CBAM attention mechanism, transfer learning, and data enhancement methods to improve the recall rate, F1 score, AP, and other indicators of YOLOX-tiny. This study improves the detection performance of YOLOX-tiny. The use of deep learning can improve measurement efficiency while ensuring measurement accuracy and stability. The results showed that the highest relative error of tree measurements was 4.06% and the average relative error was 1.62%. The analysis showed that the method performed better at all stages than in previous studies.

Investigating pedestrian-level greenery in urban forms in a high-density city for urban planning

The understanding of pedestrian-level greenery across urban forms in built environment configurations in high-density cities is insufficient. We conducted a citywide investigation of urban greenery from the pedestrian perspective by developing a deep learning technique to extract greenery from fisheye images generated from Google Street View images in Hong Kong. Relying on open-source data, we compared pedestrian-level greenery measurements with the satellite-based normalized difference vegetation index (NDVI) in diverse urban forms represented by local climate zone classes. Street greenery was spatially variant, and low greenery was found predominantly in private residential and commercial/business lands in high-density areas. Pedestrian-level measurement and the NDVI were strongly correlated, but the inconsistency between them increased from high- and mid-rise forms to low-rise forms and from compact forms to open forms. We also demonstrated the idea of integrating nearby street greenery with spatial information on population and urban morphology for inequality analysis. Potential implications for urban planning are provided. The findings linking street greenery with urban morphology are useful for urban and greenery planning in climate-resilient, sustainable, and healthy cities. Our analytical approach using open-source data is transferable to other high-density cities.

Human Tracking in Top-View Fisheye Images: Analysis of Familiar Similarity Measures via HOG and against Various Color Spaces.

The purpose of this paper is to find the best way to track human subjects in fisheye images by considering the most common similarity measures in the function of various color spaces as well as the HOG. To this end, we have relied on videos taken by a fisheye camera wherein multiple human subjects were recorded walking simultaneously, in random directions. Using an existing deep-learning method for the detection of persons in fisheye images, bounding boxes are extracted each containing information related to a single person. Consequently, each bounding box can be described by color features, usually color histograms; with the HOG relying on object shapes and contours. These descriptors do not inform the same features and they need to be evaluated in the context of tracking in top-view fisheye images. With this in perspective, a distance is computed to compare similarities between the detected bounding boxes of two consecutive frames. To do so, we are proposing a rate function () in order to compare and evaluate together the six different color spaces and six distances, and with the HOG. This function links inter-distance (i.e., the distance between the images of the same person throughout the frames of the video) with intra-distance (i.e., the distance between images of different people throughout the frames). It enables ascertaining a given feature descriptor (color or HOG) mapped to a corresponding similarity function and hence deciding the most reliable one to compute the similarity or the difference between two segmented persons. All these comparisons lead to some interesting results, as explained in the later part of the article.

VAVM: A Flexible Technique for Variable-Angle Around View Monitor System Towards Articulated Engineering Vehicle

The around view monitor system (AVM) has been widely used in passenger vehicles. However, for articulated engineering vehicles (AEV), AVM is difficult to directly apply due to the long body and the articulation angle. In this paper, a flexible technique for variable-angle around view monitor system (VAVM) towards the AEV is proposed, which includes static calibration and dynamic execution. In the static calibration stage, the camera poses in the vehicle coordinate system with the joint of AEV as the coordinate origin can be calibrated using irregularly placed multiple checkerboards without knowing their relative position. This flexible calibration approach makes the variable-angle mapping from the fisheye image to the bird’s eye view image can be directly obtained by updating the camera poses on the tractor according to the articulation angle when the AEV turns. In the dynamic execution stage, an optimization algorithm based on minimizing the photometric error is designed to reduce the cumulative error of the angle sensor. Finally, all the bird’s eye view images are mosaiced and fused to a variable-angle around view image (VAVI). The experimental results show that the VAVM can output globally balanced and seamless VAVIs under different angles and visual environment conditions. Furthermore, the practicability of VAVM is also demonstrated by a real AEV.

A comparative study of semantic segmentation of omnidirectional images from a motorcycle perspective

The semantic segmentation of omnidirectional urban driving images is a research topic that has increasingly attracted the attention of researchers, because the use of such images in driving scenes is highly relevant. However, the case of motorized two-wheelers has not been treated yet. Since the dynamics of these vehicles are very different from those of cars, we focus our study on images acquired using a motorcycle. This paper provides a thorough comparative study to show how different deep learning approaches handle omnidirectional images with different representations, including perspective, equirectangular, spherical, and fisheye, and presents the best solution to segment road scene omnidirectional images. We use in this study real perspective images, and synthetic perspective, fisheye and equirectangular images, simulated fisheye images, as well as a test set of real fisheye images. By analyzing both qualitative and quantitative results, the conclusions of this study are multiple, as it helps understand how the networks learn to deal with omnidirectional distortions. Our main findings are that models with planar convolutions give better results than the ones with spherical convolutions, and that models trained on omnidirectional representations transfer better to standard perspective images than vice versa.

A novel region-based expansion rate obstacle detection method for MAVs using a fisheye camera

Expansion-based methods are among the fastest algorithms to detect obstacles for safe navigation of Micro-Aerial Vehicles (MAVs). These methods are based on estimating an enlarging rate which is mostly computed using point features. Using points alone may result in situations where obstacles are only partially identified. This paper presents a new technique that uses image regions, instead of points, for estimating the expansion rate. The proposed algorithm utilises a fisheye camera that can be installed in front of a drone to detect obstacles in all directions. The camera takes images on which obstacles are identified. At each point in time, we extract the regions on the latest fisheye image and check to determine whether or not its regions belong to an obstacle. This step is completed by matching region points with those within the previous image. If at least three points are matched, then the convex hulls of the matched points on both images are formed. The expansion ratio of the convex hull areas is then estimated. If this ratio is bigger than a certain threshold, the region on the latest image is determined as an obstacle; otherwise, it is disregarded. This process is repeated until all pixels of the image are labelled either as obstacle or non-obstacle. Experiments were carried out using 50 pairs of fisheye images that covered a variety of obstacles, including people, pillars, trees, walls, and so on. The findings showed between 74%, and 84% of pixels were labelled correctly. By comparing these results with those obtained by the method developed by Al-Kaff et al. (2017), it is clear that the proposed method produces more stable and accurate results.

Online Correction of Camera Poses for the Surround-view System: A Sparse Direct Approach

The surround-view module is an indispensable component of a modern advanced driving assistance system. By calibrating the intrinsics and extrinsics of the surround-view cameras accurately, a top-down surround-view can be generated from raw fisheye images. However, poses of these cameras sometimes may change. At present, how to correct poses of cameras in a surround-view system online without re-calibration is still an open issue. To settle this problem, we introduce the sparse direct framework and propose a novel optimization scheme of a cascade structure. This scheme is actually composed of two levels of optimization and two corresponding photometric error based models are proposed. The model for the first-level optimization is called the ground model, as its photometric errors are measured on the ground plane. For the second level of the optimization, it’s based on the so-called ground-camera model, in which photometric errors are computed on the imaging planes. With these models, the pose correction task is formulated as a nonlinear least-squares problem to minimize photometric errors in overlapping regions of adjacent bird’s-eye-view images. With a cascade structure of these two levels of optimization, an appropriate balance between the speed and the accuracy can be achieved. Experiments show that our method can effectively eliminate the misalignment caused by cameras’ moderate pose changes in the surround-view system. Source code and test cases are available online at https://cslinzhang.github.io/CamPoseCorrection/ .

Expandable Spherical Projection and Feature Concatenation Methods for Real-Time Road Object Detection Using Fisheye Image

Fisheye lens cameras are widely used in such applications where a large field of view (FOV) is necessary. A large FOV can provide an enhanced understanding of the surrounding environment and can be an effective solution for detecting the objects in automotive applications. However, this comes with the cost of strong radial distortions and irregular size of objects depending on the location in an image. Therefore, we propose a new fisheye image warping method called Expandable Spherical Projection to expand the center and boundary regions in which smaller objects are mostly located. The proposed method produces undistorted objects especially in the image boundary and a less unwanted background in the bounding boxes. Additionally, we propose three multi-scale feature concatenation methods and provide the analysis of the influence from the three concatenation methods in a real-time object detector. Multiple fisheye image datasets are employed to demonstrate the effectiveness of the proposed projection and feature concatenation methods. From the experimental results, we find that the proposed Expandable Spherical projection and the LCat feature concatenation yield the best AP performance, which is up to 4.7% improvement compared to the original fisheye image datasets and the baseline model.

A real-time correction and stitching algorithm for underwater fisheye images

A real-time correction and stitching algorithm for underwater fisheye images

Distorted dense analogs intelligent recognition in fisheye lenses by partially featured points calibrating and clustering.

Dense analogs intelligent recognition (DAIR) has many potential applications in various fields as a new cross-disciplinary frontier of artificial intelligence and optical technology. However, with extensive application of fisheye lenses, inherent distortions in fisheye images have brought new challenges to DAIR. To solve this problem, we propose and experimentally demonstrate a partially featured points calibrating method that needs only correction of central points of the bounding boxes output by a convolutional neural network (CNN). The key to our method is a central-coordinate calibrating and clustering algorithm (CCCCA) based on a hemispheric double longitude projection model. Experimental results show that the CCCCA reduces the classification error rate by 6.05%, enhancing the classification accuracy of distorted DAIR up to 99.31%. Such classification accuracy is about 2.74% higher than that achieved by the mainstream online hard example mining algorithm, effectively modifying recognition errors induced by the CNN.

Underwater large field of view 3D imaging based on fisheye lens

As for the 3D imaging of seabed topography, seabed sunken ships and large underwater buildings, the field of view is a significant technical index of the 3D imaging system. In this paper, a binocular vision imaging system based on fisheye lens is applied to satisfy the requirements for 3D imaging of large underwater scenes. Considering the severe image distortion caused by the large wide-angle of fisheye lens and the failure of pinhole imaging model underwater, the imaging process of underwater fisheye camera is re-described through the parametric representation of four-dimensional light field, and the multi-layer refraction imaging model of underwater binocular stereo vision is established based on fisheye camera. A conversion imaging model is constructed to convert the corrected underwater fisheye image into the equivalent image taken in the air. Combined with the existing stereo matching methods, the converted images realize the 3D reconstruction of a large-scale underwater scene. Experimental results show that the corrected underwater fisheye images have a small average reprojection error, which evidences the effectiveness of the proposed calibration method. In the experiment of 3D reconstruction for the large scene composed of multiple underwater ceramic discs, the reconstructed targets have good 3D imaging shapes, which indicates the developed underwater binocular fisheye camera imaging system has the ability to perform 3D imaging and make observations of large underwater scenes.

FisheyePixPro: Self-supervised pretraining using Fisheye images for semantic segmentation

Self-supervised learning has been an active area of research in the past few years. Contrastive learning is a type of self-supervised learning method that has achieved a significant performance improvement on image classification task. However, there has been no work done in its application to fisheye images for autonomous driving. In this paper, we propose FisheyePixPro, which is an adaption of pixel level contrastive learning method PixPro \cite{Xie2021PropagateYE} for fisheye images. This is the first attempt to pretrain a contrastive learning based model, directly on fisheye images in a self-supervised approach. We evaluate the performance of learned representations on the WoodScape dataset using segmentation task. Our FisheyePixPro model achieves a 65.78 mIoU score, a significant improvement over the PixPro model. This indicates that pre-training a model on fisheye images have a better performance on a downstream task.

Estimation of Mean Radiant Temperature in Urban Canyons Using Google Street View: A Case Study on Seoul

Extreme heat exposure has severe negative impacts on humans, and the issue is exacerbated by climate change. Estimating spatial heat stress such as mean radiant temperature (MRT) is currently difficult to apply at city scale. This study constructed a method for estimating the MRT of street canyons using Google Street View (GSV) images and investigated its large-scale spatial patterns at street level. We used image segmentation using deep learning to calculate the view factor (VF) and project panorama into fisheye images. We calculated sun paths to estimate MRT using panorama images from Google Street View. This paper shows that regression analysis can be used to validate between estimated short-wave, long-wave radiation and the measurement data at seven field measurements in the clear-sky (0.97 and 0.77, respectively). Additionally, we compared the calculated MRT and land surface temperature (LST) from Landsat 8 on a city scale. As a result of investigating spatial patterns of MRT in Seoul, South Korea, we found that a high MRT of street canyons (>59.4 °C) is mainly distributed in open space areas and compact low-rise density buildings where the sky view factor is 0.6–1.0 and the building view factor (BVF) is 0.35–0.5, or west-east oriented street canyons with an SVF of 0.3–0.55. However, high-density buildings (BVF: 0.4–0.6) or high-density tree areas (Tree View Factor, TVF: 0.6–0.99) showed low MRT (<47.6). The mapped MRT results had a similar spatial distribution to the LST; however, the MRT was lower than the LST in low tree density or low-rise high-density building areas. The method proposed in this study is suitable for a complex urban environment consisting of buildings, trees, and streets. This will help decision makers understand spatial patterns of heat stress at the street level.

Global Convolutional Neural Networks With Self-Attention for Fisheye Image Rectification

Fisheye images are attracting attention in computer vision such as autonomous vehicles and virtual reality because of the wide field of view (WFoV). However, fisheye images have geometric distortions caused by the refractive index of the lens. Conventional fisheye rectification methods require multiple images to calculate distortion coefficients and lens intrinsic parameters. This means that if the fisheye lens is changed, the same operation will have to be repeated. On the other hand, by using deep learning, images with different distortion coefficients can be rectified. Also, with end-to-end learning, no feature engineering is required. To improve the performance of fisheye image rectification, we propose global convolutional neural networks with self-attention to rectify the fisheye images. The proposed method employs dilated convolutional neural networks (D-CNNs) to enlarge receptive fields, and self-attention to extract the most important features of input images. In this way, the proposed method can extract global features from input images. To better train and evaluate the proposed method, we generate fisheye images from the Place2 dataset with Cartesian and polar coordinates, and label them with original images (ground-truth). we also schedule the learning rate with cosine annealing and use an integrated loss function. The experimental results show that the proposed method achieves an excellent performance in both qualitative and quantitative evaluations.

A Stronger Stitching Algorithm for Fisheye Images Based on Deblurring and Registration

A Stronger Stitching Algorithm for Fisheye Images Based on Deblurring and Registration

A Stronger Stitching Algorithm for Fisheye Images Based on Deblurring and Registration

A Stronger Stitching Algorithm for Fisheye Images Based on Deblurring and Registration

Design and implementation of fish freshness detection algorithm using deep learning

Organoleptic assessment of fresh fish includes specifications for the quality of the eyes, gills, mucus, odor, texture and flesh (color and appearance). However, not everyone has knowledge about it. This research uses the tiny yolov2 to facilitate the determination of fish freshness levels (good quality, medium quality, poor quality) correctly and fast. There are a few stages in this research, included organoleptic test accompanied by taking fish eye image dataset every hour, processing organoleptic test data labeling, training, and validation. There are three types of fish used, consists of Rastrelliger, Euthynnus affinis, and Chanos chanos. Detection of fish freshness level for three species was successfully carried out with the result of average precision is 72.9%, average recall is 57.5%, and accuracy is 57.5%. The factors that affect the prediction results in this study is the collection of datasets before the training process is carried out consisting of fish samples obtained from traditional markets, which are considered inadequate so that it affects the organoleptic test process itself, the organoleptic test that was carried out as a reference for image sorting was considered inaccurate because it used less than 30 untrained panelists and dataset imbalance.

Efficient probability-oriented feature matching using wide field-of-view imaging

Feature matching is a key technique for a wide variety of computer vision and image processing applications such as visual localization. It permits finding correspondences of significant points within the environment that eventually determine the localization of a mobile agent. In this context, this work evaluates an Adaptive Probability-Oriented Feature Matching (APOFM) method that dynamically models the visual knowledge of the environment in terms of the probability of existence of features. Several improvements are proposed to achieve a more robust matching in a visual odometry framework: a study on the classification of the matching candidates, enhanced by a nearest neighbour search policy; a dynamic weighted matching that exploits the probability of feature existence in order to tune the matching thresholds; and an automatic false positive detector. Additionally, a comparison of performance is carried out, considering a publicly available dataset composed of two kinds of wide field-of-view images: catadioptric and fisheye. Overall, the results validate the appropriateness of these contributions, which outperform other well-recognized implementations within this framework, such as the standard visual odometry, a visual odometry method based on RANSAC, as well as the basic APOFM. The analysis shows that fisheye images provide more visual information of the scene, with more feature candidates. Contrarily, omnidirectional images produce fewer feature candidates, but with higher ratios of feature acceptance. Finally, it is concluded that improved precision is obtained when the location problem is solved by this method.

A simplified evaluation method of rooftop solar energy potential based on image semantic segmentation of urban streetscapes

The evaluation of rooftop solar energy potential in cities has a fundamental role in the development and utilization of solar energy. The irradiance-based approach of evaluation is a repetitive calculation process combined with the complex geometry of urban forms. This research paper proposes a framework that utilizes urban streetscapes to quickly estimate the urban solar energy potential using a simplified solar energy yield model based on two indicators: sky view factor (SVF) and sun coverage factor (SCF). First, 327 Google Street View images were converted into fisheye images, which were semantically segmented based on the deep-learning Full Convolutional Network (FCN). Then, the SVF and SCF were calculated based on the method of pixel statistics. Finally, a model with the simulated solar energy yield was established. Furthermore, a validation study was conducted using rooftop solar production data, which showed that the maximum estimation error of the proposed model was less than 8% and the average relative error was 6.6%. Generally, the proposed model reduced the required computation time while preserving a satisfactory degree of accuracy.

DQN-based gradual fisheye image rectification

Fisheye image rectification is a classic and important task in the computer vision area, which is generally treated as a pre-processing step in many application scenarios. Most of the existing fisheye image rectification methods focused mainly on building a direct (one-step) projection relationship between fisheye images and corrected images. Although these methods have achieved impressive performance, they depended heavily on data distribution and cannot work well on images whose distortion parameters are out of range. To address this issue, we propose a multi-step gradual image rectification scheme. In particular, we treat the fisheye image rectification problem as one Markov Decision Process and employ a widely-used deep reinforcement learning method (i.e., Deep Q-Network) to solve the problem. In addition, we build a new large-scale fisheye image dataset to evaluate our method, which contains 50,000 training images and 5000 test images. Experimental results show the proposed method can better handle images with a wide range of distortion.