Lens Distortion Research Articles

Automatic infrared image distortion correction of electrolytic cells based on refined edge lines

Lens distortion greatly affects electrolysis fault location based on infrared images. This paper proposes a lens distortion correction method for the infrared images of electrolytic cell groups (ECGs) based on refined edge lines in a single ECG infrared image. Firstly, morphological operators are designed for ECG edge extraction, and the Hough transform method and an improved hierarchical method are combined to automatically separated the edge lines. Then, the least squares method is applied to refine the lines, and an optimization solution model is designed to calibrate the distortion parameters according to the refined lines. At last, an interpolation strategy is presented to restore pixel values of the corrected images with high speed and quality. The proposed method obtains a root mean squared error of the corrected lines of about 0.014 pixels, which is advanced and meets the fault location need. Moreover, high effectiveness is obtained with the proposed interpolation strategy.

Automatically drawing vegetation classification maps using digital time‐lapse cameras in alpine ecosystems

AbstractAlpine ecosystems are particularly vulnerable to climate change. Monitoring the distribution of alpine vegetation is required to plan practical conservation activities. However, conventional field observations, airborne and satellite remote sensing are difficult in terms of coverage, cost and resolution in alpine areas. Ground‐based time‐lapse cameras have been used to observe the regions' snowmelt and vegetation phenology and offer significant advantages in terms of cost, resolution and frequency. However, they have not been used in research monitoring of vegetation distribution patterns. This study proposes a novel method for drawing georeferenced vegetation classification maps from ground‐based imagery of alpine regions. Our approach had two components: vegetation classification and georectification. The proposed vegetation classification method uses a pixel time series acquired from fall images, utilizing the fall leaf color patterns. We demonstrated that the performance of the vegetation classification could be improved using time‐lapse imagery and a Recurrent Neural Network. We also developed a novel method to accurately transform ground‐based images into georeferenced data. We propose the following approaches: (1) an automated procedure to acquire Ground Control Points and (2) a camera model that considers lens distortions for accurate georectification. We demonstrated that the proposed approach outperforms conventional methods, in addition to achieving sufficient accuracy to observe the vegetation distribution on a plant‐community scale. The evaluation revealed an F1 score and root‐mean‐square error of 0.937 and 3.4 m in the vegetation classification and georectification, respectively. Our results highlight the potential of inexpensive time‐lapse cameras to monitor the distribution of alpine vegetation. The proposed method can significantly contribute to the effective conservation planning of alpine ecosystems.

Predicting Subjective Discomfort Associated with Lens Distortion in VR Headsets During Vestibulo-Ocular Response to VR Scenes.

With advances in Virtual Reality (VR) technology, user expectation for a near-perfect experience is also increasing. The push for a wider field-of-view can increase the challenges of correcting lens distortion. Past studies on imperfect VR experiences have focused on motion sickness provoked by vection-inducing VR stimuli and discomfort due to mismatches in accommodation and binocular convergence. Disorientation and discomfort due to unintended optical flow induced by lens distortion, referred to as dynamic distortion (DD), has, to date, received little attention. This study examines and models the effects of DD during head rotations with various fixed gazes stabilized by vestibulo-ocular reflex (VOR). Increases in DD levels comparable to lens parameters from poorly designed commercial VR lenses significantly increase discomfort scores of viewers in relation to disorientation, dizziness, and eye strain. Cross-validated results indicate that the model is able to predict significant differences in subjective scores resulting from different commercial VR lenses and these predictions correlated with empirical data. The present work provides new insights to understand symptoms of discomfort in VR during user interactions with static world-locked / space-stabilized scenes and contributes to the design of discomfort-free VR headset lenses.

Where's your head at? Detecting the orientation and position of pigs with rotated bounding boxes

Pig detection in real production environments is a challenging task due to the variations of housing system and dynamic background. Though considerable progress has been made, for practical settings, there still existing challenges for densely housed pigs as they are often arbitrarily arranged at varying orientations in presence of lens distortion, overlap, occlusion, and motion blur. In this paper, we propose a rotated and oriented bounding box detector for fast and accurate predict the location and orientation of each animal. The key point is to parameterize pigs’ geometric parameters (body centre, body length, body width, orientation) with an orientated bounding box (box centre, long edge, short edge and direction vector). To further improve the performance on video object detection, a fast sequential non-Maximum Suppression (FastSeq-NMS) method is proposed by making used of the orientation and temporal information. To quantitative evaluate the proposed method, 3123 images from 27 different pens were selected as training and validation sets, video clips from three new environments were selected as test set. Our lightweight model (1.7 M) achieves 99.21 Average Precision (AP@0.5) on validation set, and 96.54 AP@0.5 on test set, further improved to 97.41 AP@0.5 with the proposed NMS method. The experiments show the effectiveness of the proposed method. More information available online: https://gitlab.kuleuven.be/m3-biores/public/m3pig.

Modelling and Estimation of Interior Orientation of Non-Metric Cameras using Artificial Intelligence

With the advancement of low-cost digital cameras and their widespread, especially in smartphones, these cameras are not designed for photogrammetry applications. The main aim of this study is to model and estimate the interior orientation parameters (IOPs) for images captured by volunteer-run cameras using artificial intelligence. These cameras were unavailable to perform traditional calibration processes or use images from unknown sources for image measuring. Estimating IOPs using random values within the range determined by testing the selected sample's consistency. Optimization was performed by Utilizing the Simulated Annealing algorithm based on stereo calibration to obtain the best possible values of these parameters that produce the minimum RMS-reprojection error attained. By The variance between the parameters from the pre-calibration process and estimated by an artificial intelligence system, The coefficient of determination in the IOPs (focal length R2 = 0.717 to 0.812, principal point (X, Y) R2 = (0.674 to 0.869, 0.504 to 0.613), Both radial and tangential coefficients, R2 was close to zero). Therefore, the estimations of radial distortions k1, k2, and tangential distortions p1 and p2 are invalid. A reasonably strong relationship between principal distance and principal point with low lens distortion parameters due to the significant relative differences between the distortion parameters, sufficient strength of the relationship between calculating parameters, and estimating according to accuracy tolerance in photogrammetry applications.

An improved checkerboard detection algorithm based on adaptive filters

Checkerboard corner extraction is a crucial step in camera calibration. However, most existing algorithms are not good enough if the lens distortion is too large. This study aims to propose a checkerboard corner detection algorithm based on adaptive filters to address this problem. First, adaptive filters based on local image features are used to generate response maps of checkerboard images. Next, a non-max suppression and a scoring system are applied for further screening. Finally, the whole checkerboard structure is restored via inertia growth. The algorithm proposed is subject to rigorous experimental validation using synthetic and real images. Compared with several state of the art methods, our algorithm has the best performance.

The development of rowing performance evaluation technique using high-speed imaging

Rowing as an on-river sports has a long history dating back to the 16th century. The deeper understanding of fluid dynamics theory and the development of advanced material and manufacturing science and technology have inevitably fueled the continuous improvement on the world record of the rowing sports. The current paper focuses on the accurate extraction of the kinematics information including the boats location, velocity, and acceleration, resulting from the athletes rowing motion. A consumer-level digital camera is used for imaging the rowing motion at 50 frames per second from a distance, corresponding to a specific field of view ranging from 12 m 36 m. Furthermore, due to large-scale imaging lens distortion, the physical resolution is varying significantly due to imaging location change. A video-specific calibration method using fourth order polynomials are developed to calibrate the physical resolution of each pixel based on the physical length of the boat. The resulting data is pixel-level accurate and provides detailed record for athletes training. The acquired information is extremely useful in gauging athletes performance and can be further analyzed to provide concrete technical improvement advice.

Pixel-wise rational model for a structured light system.

This Letter presents a novel structured light system model that effectively considers local lens distortion by pixel-wise rational functions. We leverage the stereo method for initial calibration and then estimate the rational model for each pixel. Our proposed model can achieve high measurement accuracy within and outside the calibration volume, demonstrating its robustness and accuracy.

A stable and effective calibration method for defocused cameras using synthetic speckle patterns

Well-focused and sharp target images are essential in traditional calibration, but they generally cause difficulty and inconvenience in image acquisition in practical applications. To address this problem, a stable and effective calibration method for defocused cameras is proposed in this study. Firstly, two speckle patterns are created, one of which has control points to estimate the initial value while the other, used for point pair match registration, is without control points. They are subsequently displayed on a fixed monitor and captured at each camera pose. Secondly, the initial guess for the control points is estimated using the proposed method. The retrieval of query points can then be done with high accuracy using the digital image correlation algorithm. Finally, all the calibration parameters, including lens distortion, are calculated using the plane calibration algorithm. The experiment in this study demonstrates the practicality and effectiveness of the proposed method. The accuracy and precision were quantitatively evaluated by the mean reprojection error, which was 0.057 pixels.

Calibration method for fringe projection profilometry based on rational function lens distortion model

The calibration of the fringe projection profilometry (FPP) system is susceptible to the camera lens distortion. This paper presents a calibration method based on the rational function lens distortion model. In this method, we integrate the rational function into the mathematical expression of the out-of-plane height to mitigate the impact of lens distortion. With arbitrarily arranged measurement components, we can flexibly measure the three-dimensional (3D) shape of objects with no expensive auxiliaries, high-precision setups, or complicated projector calibration. First, we project modulated sinusoidal fringe patterns with high frequency to the concentric circle grids calibration board to accurately estimate the unwrapped phase value, the pixel coordinate, and the out-of-plane height at each feature point. Then, the 3D coordinates can be recovered by determining the coefficients of the high order phase, coordinate, and height (PCH) mapping. Experiments on computer simulation and real data demonstrate that the modified calibration contributes to the flexibility and accuracy of the 3D shape measurement.

English

The estimation of the depth of a scene has become a research niche due to the large number of applications that exist today. In a stereo vision system, disparity maps make it possible to obtain the depth of a scene from two rectified images. Stereo vision systems are sensitive to illumination, reflections, lens distortion, noise, camera alignment, etc.; therefore, in this work, we present an algorithm that allows us to improve the disparity map when the environment presents particles in suspension using stereo vision through infrared cameras. For this purpose, filters, matching, and dehazing algorithms are implemented.

All-parameter calibration method of the on-orbit multi-view dynamic photogrammetry system.

Photogrammetry (PG) can present accurate data to evaluate the functional performance of large space structures. For camera calibration and orientation, the On-orbit Multi-view Dynamic Photogrammetry System (OMDPS) lacks appropriate spatial reference data. A multi-data fusion calibration method for all parameters for this kind of system is proposed in this paper as a solution to this issue. Firstly, a multi-camera relative position model is developed to solve the reference camera position unconstrained problem in the full-parameter calibration model of the OMDPS in accordance with the imaging model of stars and scale bar targets. Subsequently, the problem of adjustment failure and inaccurate adjustment in the multi-data fusion bundle adjustment is solved using the two-norm matrix and the weight matrix to adjust the Jacobian matrix with respect to all system parameters (e.g., camera interior parameters (CIP), camera exterior parameters (CEP), and lens distortion parameters (LDP)). Finally, all system parameters can be optimized simultaneously using this algorithm. In the actual data ground-based experiment, 333 spatial targets are measured using the V-star System (VS) and OMDPS. Taking the measurement of VS as the true value, the measurement results of OMDPS indicated that the in-plane Z-direction target coordinates root-mean-square error (RMSE) is less than 0.0538 mm and the Z-direction RMSE is less than 0.0428 mm. Out-of-plane Y-direction RMSE is less than 0.1514 mm. The application potential of the PG system for on-orbit measurement tasks is demonstrated through the actual data ground-based experiment.

Decoupling calibration method of long focal length camera using vanishing point

Optical measurement and perception technology is widely used in the field of smart energy. The accurate calibration of the internal and external parameters of the camera in the optical system is very important for the application of the system in three-dimensional (3D) reconstruction and geometric measurement. At present, the mainstream camera calibration methods include Zhang’s calibration method and Tsai’s calibration method. These methods all choose to calculate the distortion parameters together with the camera’s internal parameters. For long focal length and narrow-field-of-view cameras with smaller perspective distortion, the coupling calculation of parameters may cause inaccurate calibration parameters and more time-consuming problems. To improve the calibration accuracy of the long focal length camera, we propose an efficient noniterative camera calibration method, based on the equation relationship between the vanishing point coordinates and the first-order single-parameter division lens distortion coefficient, and based on the radial distortion separation model as well as the corner subpixel coordinates and checkerboard 3D space points. The spatial point correspondence is solved to obtain the homography matrix to complete the calculation of the internal parameters of the camera. Our work has potential applications in photovoltaic troubleshooting and intelligent inspection. It may also contribute to the practical application of the sensor in intelligent energy.

Estimation of Plate and Bowl Dimensions for Food Portion Size Assessment in a Wearable Sensor System.

Automatic food portion size estimation (FPSE) with minimal user burden is a challenging task. Most of the existing FPSE methods use fiducial markers and/or virtual models as dimensional references. An alternative approach is to estimate the dimensions of the eating containers prior to estimating the portion size. In this article, we propose a wearable sensor system (the automatic ingestion monitor integrated with a ranging sensor) and a related method for the estimation of dimensions of plates and bowls. The contributions of this study are: 1) the model eliminates the need for fiducial markers; 2) the camera system [automatic ingestion monitor version 2 (AIM-2)] is not restricted in terms of positioning relative to the food item; 3) our model accounts for radial lens distortion caused due to lens aberrations; 4) a ranging sensor directly gives the distance between the sensor and the eating surface; 5) the model is not restricted to circular plates; and 6) the proposed system implements a passive method that can be used for assessment of container dimensions with minimum user interaction. The error rates (mean ± std. dev) for dimension estimation were 2.01% ± 4.10% for plate widths/diameters, 2.75% ± 38.11% for bowl heights, and 4.58% ± 6.78% for bowl diameters.

Road User Position and Speed Estimation via Deep Learning from Calibrated Fisheye Videos.

In this paper, we present a deep learning processing flow aimed at Advanced Driving Assistance Systems (ADASs) for urban road users. We use a fine analysis of the optical setup of a fisheye camera and present a detailed procedure to obtain Global Navigation Satellite System (GNSS) coordinates along with the speed of the moving objects. The camera to world transform incorporates the lens distortion function. YOLOv4, re-trained with ortho-photographic fisheye images, provides road user detection. All the information extracted from the image by our system represents a small payload and can easily be broadcast to the road users. The results show that our system is able to properly classify and localize the detected objects in real time, even in low-light-illumination conditions. For an effective observation area of 20 m × 50 m, the error of the localization is in the order of one meter. Although an estimation of the velocities of the detected objects is carried out by offline processing with the FlowNet2 algorithm, the accuracy is quite good, with an error below one meter per second for urban speed range (0 to 15 m/s). Moreover, the almost ortho-photographic configuration of the imaging system ensures that the anonymity of all street users is guaranteed.

Flexible and fast calibration method for uni-directional multi-line structured light system

Multi-line Shift Profilometry (MLSP) is widely used to measure the high-reflectivity object by projecting high-contrast binary stripe image. However, in the calibration of a MLSP system, the lens distortion of the projector makes the most existing calibration methods fail in obtaining high accurate result. Therefore, to avoid the influence of lens distortion, we propose a novel flexible and fast calibration method for MLSP, which requiring only uni-directional patterns. In this method, for each bending light plane, lens distortion is avoided by establishing a polynomial model between three-dimensional (3D) coordinates and camera sub-pixel coordinates. Because the position of calibration planar targets is placed freely without any constraint, this method is flexible and convenient to operate. Experimental results demonstrate that the proposed method achieves high-precision 3D reconstruction.

A Combined Physical and Mathematical Calibration Method for Low-Cost Cameras in the Air and Underwater Environment.

Low-cost camera calibration is vital in air and underwater photogrammetric applications. However, various lens distortions and the underwater environment influence are difficult to be covered by a universal distortion compensation model, and the residual distortions may still remain after conventional calibration. In this paper, we propose a combined physical and mathematical camera calibration method for low-cost cameras, which can adapt to both in-air and underwater environments. The commonly used physical distortion models are integrated to describe the image distortions. The combination is a high-order polynomial, which can be considered as basis functions to successively approximate the image deformation from the point of view of mathematical approximation. The calibration process is repeated until certain criteria are met and the distortions are reduced to a minimum. At the end, several sets of distortion parameters and stable camera interior orientation (IO) parameters act as the final camera calibration results. The Canon and GoPro in-air calibration experiments show that GoPro owns distortions seven times larger than Canon. Most Canon distortions have been described with the Australis model, while most decentering distortions for GoPro still exist. Using the proposed method, all the Canon and GoPro distortions are decreased to close to 0 after four calibrations. Meanwhile, the stable camera IO parameters are obtained. The GoPro Hero 5 Black underwater calibration indicates that four sets of distortion parameters and stable camera IO parameters are obtained after four calibrations. The camera calibration results show a difference between the underwater environment and air owing to the refractive and asymmetric environment effects. In summary, the proposed method improves the accuracy compared with the conventional method, which could be a flexible way to calibrate low-cost cameras for high accurate in-air and underwater measurement and 3D modeling applications.

Deep learning: survey of environmental and camera impacts on internet of things images

Internet of Things (IoT) images are captivating growing attention because of their wide range of applications which requires visual analysis to drive automation. However, IoT images are predominantly captured from outdoor environments and thus are inherently impacted by the camera and environmental parameters which can adversely affect corresponding applications. Deep Learning (DL) has been widely adopted in the field of image processing and computer vision and can reduce the impact of these parameters on IoT images. Albeit, there are many DL-based techniques available in the current literature for analyzing and reducing the environmental and camera impacts on IoT images. However, to the best of our knowledge, no survey paper presents state-of-the-art DL-based approaches for this purpose. Motivated by this, for the first time, we present a Systematic Literature Review (SLR) of existing DL techniques available for analyzing and reducing environmental and camera lens impacts on IoT images. As part of this SLR, firstly, we reiterate and highlight the significance of IoT images in their respective applications. Secondly, we describe the DL techniques employed for assessing the environmental and camera lens distortion impacts on IoT images. Thirdly, we illustrate how DL can be effective in reducing the impact of environmental and camera lens distortion in IoT images. Finally, along with the critical reflection on the advantages and limitations of the techniques, we also present ways to address the research challenges of existing techniques and identify some further researches to advance the relevant research areas.

Distortion Detection of Lithographic Projection Lenses Based on Wavefront Measurement

As with the decreasing feature size prompted by Moore’s law and the continuous technological advancements in the semiconductor industry, the distortion of the projection lens is an important factor that affects the overlay. In this paper, we propose a wavefront-measurement-based method to detect the projection lens distortion in the lithographic system. By normalizing the coordinates of the Shack–Hartmann system with the image displacements represented in the Z2 and Z3 terms of Zernike coefficients, the offsets between the actual image points and the ideal image points can be determined. By offset collection at an array of 7 × 7 field points to establish an overdetermined system of equations, the proposed method can simultaneously detect the distortions of translation, magnification, rotation, decentering distortion, thin prism distortion, and third-order radial distortion. This distortion measurement method is highly flexible for distortion measurement with portable and compactly integrated sensors, enabling the real-time and cost-efficient measurement of wave aberration and distortion. For proof-of-concept experiments, a projection lens with a numerical aperture (NA) of 0.58 for i-line (365 nm) is used for experimental testing. The results reveal that the repeatability accuracy of distortion detection is 51 nm and the 72 h long-term reproducibility is 143 nm.

Automated rolling shutter calibration with an LED panel.

Cameras with rolling shutters (RSs) dominate consumer markets but are subject to distortions when capturing motion. Many methods have been proposed to mitigate RS distortions for applications such as vision-aided odometry and three-dimensional (3D) reconstruction. They usually need known line delay d between successive image rows. To calibrate d, several methods have been proposed that often involve complex procedures. This Letter proposes an easy RS calibration method by using an off-the-shelf light-emitting diode (LED) panel, using the fact that the RS causes the blinking LED columns to appear slanted in images by a static camera. The calibration starts with extracting the LED lights and then rectifies the images to remove the lens distortion and misalignment between the camera and the LED panel. Next, blocks of slanted bright LEDs are recognized and their inclination leads to the line delay estimate. Our method needs not to move the camera, adjust the ambient light, or calibrate camera intrinsic parameters beforehand, and it can usually estimate the line delay given two LED panel images in one second. Extensive tests with industrial cameras and consumer cameras of wide-angle and fish-eye lenses validate its competitive accuracy relative to the established methods.