Fisheye Projection Research Articles

Omni-OTPE: Omnidirectional Optimal Real-Time Ground Target Position Estimation System for Moving Lightweight Unmanned Aerial Vehicle.

Ground target detection and positioning systems based on lightweight unmanned aerial vehicles (UAVs) are increasing in value for aerial reconnaissance and surveillance. However, the current method for estimating the target's position is limited by the field of view angle, rendering it challenging to fulfill the demands of a real-time omnidirectional reconnaissance operation. To address this issue, we propose an Omnidirectional Optimal Real-Time Ground Target Position Estimation System (Omni-OTPE) that utilizes a fisheye camera and LiDAR sensors. The object of interest is first identified in the fisheye image, and then, the image-based target position is obtained by solving using the fisheye projection model and the target center extraction algorithm based on the detected edge information. Next, the LiDAR's real-time point cloud data are filtered based on position-direction constraints using the image-based target position information. This step allows for the determination of point cloud clusters that are relevant to the characterization of the target's position information. Finally, the target positions obtained from the two methods are fused using an optimal Kalman fuser to obtain the optimal target position information. In order to evaluate the positioning accuracy, we designed a hardware and software setup, mounted on a lightweight UAV, and tested it in a real scenario. The experimental results validate that our method exhibits significant advantages over traditional methods and achieves a real-time high-performance ground target position estimation function.

Generating a full spherical view by modeling the relation between two fisheye images

Full spherical views provide advantages in many applications that use visual information. Dual back-to-back fisheye cameras are receiving much attention to obtain this type of view. However, obtaining a high-quality full spherical view is very challenging. In this paper, we propose a correction step that models the relation between the pixels of the pair of fisheye images in polar coordinates. This correction is implemented during the mapping from the unit sphere to the fisheye image using the equidistant fisheye projection. The objective is that the projections of the same point in the pair of images have the same position on the unit sphere after the correction. In this way, they will also have the same position on the equirectangular coordinate system. Consequently, the discontinuity between the spherical views for blending is minimized. Throughout the manuscript, we show that the angular polar coordinates of the same scene point in the fisheye images are related by a sine function and the radial distance coordinates by a linear function. Also, we propose employing a polynomial as a geometric transformation between the pair of spherical views during the image alignment since the relationship between the matching points of pairs of spherical views is not linear, especially in the top/bottom regions. Quantitative evaluations demonstrate that using the correction step improves the quality of the full spherical view, i.e. IQ MS-SSIM, up to 7%. Similarly, using a polynomial improves the IQ MS-SSIM up to 6.29% with respect to using an affine matrix.

FisheyeHDK: Hyperbolic Deformable Kernel Learning for Ultra-Wide Field-of-View Image Recognition

Conventional convolution neural networks (CNNs) trained on narrow Field-of-View (FoV) images are the state-of-the art approaches for object recognition tasks. Some methods proposed the adaptation of CNNs to ultra-wide FoV images by learning deformable kernels. However, they are limited by the Euclidean geometry and their accuracy degrades under strong distortions caused by fisheye projections. In this work, we demonstrate that learning the shape of convolution kernels in non-Euclidean spaces is better than existing deformable kernel methods. In particular, we propose a new approach that learns deformable kernel parameters (positions) in hyperbolic space. FisheyeHDK is a hybrid CNN architecture combining hyperbolic and Euclidean convolution layers for positions and features learning. First, we provide intuition of hyperbolic space for wide FoV images. Using synthetic distortion profiles, we demonstrate the effectiveness of our approach. We select two datasets - Cityscapes and BDD100K 2020 - of perspective images which we transform to fisheye equivalents at different scaling factors (analogue to focal lengths). Finally, we provide an experiment on data collected by a real fisheye camera. Validations and experiments show that our approach improves existing deformable kernel methods for CNN adaptation on fisheye images.

Efficient Face Detection in the Fisheye Image Domain.

Significant progress has been made for face detection from normal images in recent years; however, accurate and fast face detection from fisheye images remains a challenging issue because of serious fisheye distortion in the peripheral region of the image. To improve face detection accuracy, we propose a light-weight location-aware network to distinguish the peripheral region from the central region in the feature learning stage. To match the face detector, the shape and scale of the anchor (bounding box) is made location dependent. The overall face detection system performs directly in the fisheye image domain without rectification and calibration and hence is agnostic of the fisheye projection parameters. Experiments on Wider-360 and real-world fisheye images using a single CPU core indeed show that our method is superior to the state-of-the-art real-time face detector RFB Net.

Deep Face Rectification for 360° Dual-Fisheye Cameras.

Rectilinear face recognition models suffer from severe performance degradation when applied to fisheye images captured by 360° back-to-back dual fisheye cameras. We propose a novel face rectification method to combat the effect of fisheye image distortion on face recognition. The method consists of a classification network and a restoration network specifically designed to handle the non-linear property of fisheye projection. The classification network classifies an input fisheye image according to its distortion level. The restoration network takes a distorted image as input and restores the rectilinear geometric structure of the face. The performance of the proposed method is tested on an end-to-end face recognition system constructed by integrating the proposed rectification method with a conventional rectilinear face recognition system. The face verification accuracy of the integrated system is 99.18% when tested on images in the synthetic Labeled Faces in the Wild (LFW) dataset and 95.70% for images in a real image dataset, resulting in an average accuracy improvement of 6.57% over the conventional face recognition system. For face identification, the average improvement over the conventional face recognition system is 4.51%.

Focus+context map labeling with optimized clutter reduction

ABSTRACTZooming is a basic operation that many digital maps support for exploring them interactively. Especially for maps on small screen devices, this is a helpful operation to uncover the user's region of interest possibly hidden by labels, e.g. points of interest represented by icons. However, by scaling the map larger the user loses the context. As a consequent the user might need to repeatedly zoom in and out to explore the map step by step.We present an approach that reduces the necessity of zooming by providing the user with the possibility of displacing the labels of a circular focus region. To that end, we utilize techniques from focus+context maps implementing the displacement of the labels by fish-eye projections. The visual association between labels and their point features is established by connecting lines aggregated to bundles. Our approach particularly guarantees that labels move smoothly when the user continuously shifts the focus region, which reduces distracting flickering effects while exploring the map by panning the map view. Further, when the user stops moving the focus region, mathematical programming is applied to optimize positions of the displaced labels. In an evaluation on real-world data and synthetically generated data, we show that our approach substantially increases the legibility of both the focus region and the displaced labels.

LaFiDa—A Laserscanner Multi-Fisheye Camera Dataset

In this article, the Laserscanner Multi-Fisheye Camera Dataset (LaFiDa) for applying benchmarks is presented. A head-mounted multi-fisheye camera system combined with a mobile laserscanner was utilized to capture the benchmark datasets. Besides this, accurate six degrees of freedom (6 DoF) ground truth poses were obtained from a motion capture system with a sampling rate of 360 Hz. Multiple sequences were recorded in an indoor and outdoor environment, comprising different motion characteristics, lighting conditions, and scene dynamics. The provided sequences consist of images from three—by hardware trigger—fully synchronized fisheye cameras combined with a mobile laserscanner on the same platform. In total, six trajectories are provided. Each trajectory also comprises intrinsic and extrinsic calibration parameters and related measurements for all sensors. Furthermore, we generalize the most common toolbox for an extrinsic laserscanner to camera calibration to work with arbitrary central cameras, such as omnidirectional or fisheye projections. The benchmark dataset is available online released under the Creative Commons Attributions Licence (CC-BY 4.0), and it contains raw sensor data and specifications like timestamps, calibration, and evaluation scripts. The provided dataset can be used for multi-fisheye camera and/or laserscanner simultaneous localization and mapping (SLAM).

Foveated fisheye lens design using an angle-variant distortion projection function

Light projection function is a major area of interest within the field of designing ultra-wide angle cameras. In this paper, we introduce a novel ultra-wide angle projection function that is characterized by an angle-variant distortion model similar to that of human retina space-variant resolution. The projection peculiarities were compared with the classical equidistant fisheye projection function to illustrate the benefits of our projection model on real-time tracking and ultra-wide angle imaging applications. The new projection model produced an accurate result with uncomplicated distortion control using Zemax user-defined macro program. The inverted model is also successfully used in correcting such distorted image. The paper shows as well the design of an original 170° fast foveated fisheye lens that provides more than 52 % undistorted image with a high-resolution performance over the entire lens field of view.

Using urban canyon geometries obtained from Google Street View for atmospheric studies: Potential applications in the calculation of street level total shortwave irradiances

This paper presents an implementation of digital image processing tools, in order to reconstruct building geometries and urban sky view factors for individual street canyons around the world. By using the Google Street View image database, it is possible to reconstruct canyon fisheye images at any point where a Google panorama is available. The Hugin software allows the geometrically correct merging of independent images in order to create a fisheye projection, covering the upper hemispheric view of the sky. Sky view factor calculations may then be performed by specialised software on the reconstructed fisheye image. In order to analyse the fitness of reconstructed panoramas, control images taken with a fisheye lens camera were used for comparison. The Rayman model was then used to calculate total shortwave global irradiances from the restricted canyon view and from a full hemispheric view. Finally, total shortwave global irradiances were measured for the same position in the street canyon and compared against observatory measurements under a full sky view, this in order to evaluate the performance and usefulness of idealised calculations. The reconstructed urban geometries returned acceptable fine-scale descriptions, and their potential utility for scientific purposes in the atmospheric sciences was demonstrated with the examples provided.

High-Accuracy Angle Detection for the Ultrawide FOV Acquisition Scheme in Free-Space Optical Links

To achieve rapid establishment of free-space optical (FSO) links, we previously proposed an ultrawide field-of-view (FOV) acquisition scheme, which combines the fisheye lens and Voigt anomalous dispersion optical filter. In this paper, we focus on the angle detection accuracy of this ultrawide FOV acquisition scheme, which is one of primary factors for rapid establishment of FSO links. The positioning process is composed of two parts. One is the spot location, which acquires the position of the imaging spot of the incident beam. The other is the mapping from spot location to incident angle, with which the position of imaging spot can be turned into the angles of the incident beam. For the first part, an improved centroid locating algorithm is present to realize the subpixel accuracy of spot location, and theory and simulation are carried out to demonstrate the improved accuracy of this algorithm. For the second part, we propose the fisheye lens calibration algorithm, which integrates the fisheye lens projection model and natural neighbor interpolation algorithm. In addition, by combining with the measured data, the angle detection accuracy of this method can be improved gradually with the increased number of the measured data. The simulation results indicate that this new method can provide high-precision positioning of the laser beams, and the maximum angle detection error of this ultrawide FOV acquisition scheme can be less than 0.054° (0.94 mrad).

3D to 2D bijection for spherical objects under equidistant fisheye projection

The core problem addressed in this article is the 3D position detection of a spherical object of known-radius in a single image frame, obtained by a dioptric vision system consisting of only one fisheye lens camera that follows equidistant projection model. The central contribution is a bijection principle between a known-radius spherical object’s 3D world position and its 2D projected image curve, that we prove, thus establishing that for every possible 3D world position of the spherical object, there exists a unique curve on the image plane if the object is projected through a fisheye lens that follows equidistant projection model. Additionally, we present a setup for the experimental verification of the principle’s correctness. In previously published works we have applied this principle to detect and subsequently track a known-radius spherical object.

Interactive focus maps using least-squares optimization

We present a new algorithm that enlarges a focus region in a given network map without removing non-focus (i.e., context) network parts from the map or changing the map’s size. In cartography, this problem is usually tackled with fish-eye projections, which, however, introduce severe distortion. Our new algorithm minimizes distortion and, with respect to this objective, produces results of similar quality compared to an existing algorithm. In contrast to the existing algorithm, the new algorithm achieves real-time performance that allows its application in interactive systems. We target applications where a user sets a focus by brushing parts of the network or the focus region is defined as the neighborhood of a moving user.A crucial feature of the algorithm is its capability of avoiding unwanted edge crossings. Basically, we first solve a least-squares optimization problem without constraints for avoiding edge crossings. The solution we find is then used to identify a small set of constraints needed for a crossing-free solution and, beyond this, allows us to start an animation enlarging the focus region before the final, crossing-free solution is found. Moreover, memorizing the non-crossing constraints from an initial run of the algorithm allows us to achieve a better runtime on any further run – assuming that the focus region does not move too much between two consecutive runs. As we show with experiments on real-world data, this enables response times well below 1 second.

Comparing artistic and geometrical perspective depictions of space in the visual field.

Which is the most accurate way to depict space in our visual field? Linear perspective, a form of geometrical perspective, has traditionally been regarded as the correct method of depicting visual space. But artists have often found it is limited in the angle of view it can depict; wide-angle scenes require uncomfortably close picture viewing distances or impractical degrees of enlargement to be seen properly. Other forms of geometrical perspective, such as fisheye projections, can represent wider views but typically produce pictures in which objects appear distorted. In this study we created an artistic rendering of a hemispherical visual space that encompassed the full visual field. We compared it to a number of geometrical perspective projections of the same space by asking participants to rate which best matched their visual experience. We found the artistic rendering performed significantly better than the geometrically generated projections.

Equisolid Fisheye Stereovision Calibration and Point Cloud Computation

Abstract. This paper deals with dense 3D point cloud computation of urban environments around a vehicle. The idea is to use two fisheye views to get 3D coordinates of the surrounding scene's points. The first contribution of this paper is the adaptation of an omnidirectional stereovision self-calibration algorithm to an equisolid fisheye projection model. The second contribution is the description of a new epipolar matching based on a scan-circle principle and a dynamic programming technique adapted for fisheye images. The method is validated using both synthetic images for which ground truth is available and real images of an urban scene.

Drawing Road Networks with Focus Regions

Mobile users of maps typically need detailed information about their surroundings plus some context information about remote places. In order to avoid that the map partly gets too dense, cartographers have designed mapping functions that enlarge a user-defined focus region--such functions are sometimes called fish-eye projections. The extra map space occupied by the enlarged focus region is compensated by distorting other parts of the map. We argue that, in a map showing a network of roads relevant to the user, distortion should preferably take place in those areas where the network is sparse. Therefore, we do not apply a predefined mapping function. Instead, we consider the road network as a graph whose edges are the road segments. We compute a new spatial mapping with a graph-based optimization approach, minimizing the square sum of distortions at edges. Our optimization method is based on a convex quadratic program (CQP); CQPs can be solved in polynomial time. Important requirements on the output map are expressed as linear inequalities. In particular, we show how to forbid edge crossings. We have implemented our method in a prototype tool. For instances of different sizes, our method generated output maps that were far less distorted than those generated with a predefined fish-eye projection. Future work is needed to automate the selection of roads relevant to the user. Furthermore, we aim at fast heuristics for application in real-time systems.

Planar motion and three-dimensional structure recovery from image sequences based on a fish-eye projection model

In this paper we propose a factorization method that can simultaneously recover planar motion and three-dimensional structure from image sequences captured using a fish-eye camera. Here we assume t...

3Desque: interface elements for a 3D graphical user interface

This paper explores some design possibilities and constraints for 3D enhancements of graphical user interfaces (GUIs). This is done with the aim of conveying more information in less screen space, while avoiding visual clutter. Design elements include the use of ‘slapped-back’ windows obtained by projecting windows and icons into three dimensions; ‘trays’, an alternative to folders or piles; the use of transparent ‘beams’ for indicating hierarchy; and the use of ‘periphs’ for a form of fisheye projection on window borders. In addition we employ shading and shadows to enhance the interface, and automatic placement algorithms to prevent visually confusing occlusion. By considering these approaches together, rather than in isolation, the interactions between the different modifications are made explicit. Copyright © 1999 John Wiley & Sons, Ltd.

The Tucson 35-mm Fisheye Projection Lens

The Tucson 35-mm Fisheye Projection Lens

視的配景変換系の構成とその応用 (II) : ⊿_0 尺度の二次元化とその写像系の解析

This paper consists of two experimental study. First, in Experiment IIA, adjustments of subjective parallel vertical line were carried out for the purpose of investigate about perceptual homogeneousness and aptitude of a polar coordinate system (called ⊿ coordinate system), simply 2-dimensionaly extended with the one dimensional scale, constructed by the Auther in previous paper, on the assumption of homogeneous direction angle φ. From the results of multi-regression analyis for the data of estimated error on the adjustment, it is known that the ⊿ coordinate system is perceptualy homogeneous and is regarded as essentialy appropriate system. Moreover, investigating the character of adjusted parallel lines that are perspecively transformed to front parallel plane, it is shown that a hyperbola which is known as Helmholtz's chess-bord is insolvable by means of a fish-eye projection regarding the spherical surface as a functional retina. Second, in Experiment IIB, adjustments of subjective rectangle were carried out. Consequently, it is shown that a problem on character of figure in perceptual dimension is directly solved in Perspective Vision as a character of similar figure in geometric dimension. This fact shows that the hypothesis of the Perspective Vision has a clear meaning in visual space perception, and on the other hand, proving the ⊿ coordinate system is a sufficient approximation for the representation of Perspective Vision. Further, it is suggested that some anisotropy for the ⊿ coordinate system and some visual constancy for judging the parallelness should be latently existed.