Interior Orientation Research Articles

Vision-guided robot calibration using photogrammetric methods

We propose novel photogrammetry-based robot calibration methods for industrial robots that are guided by cameras or 3D sensors. Compared to state-of-the-art methods, our methods are capable of calibrating the robot kinematics, the hand–eye transformations, and, for camera-guided robots, the interior orientation of the camera simultaneously. Our approach uses a minimal parameterization of the robot kinematics and hand–eye transformations. Furthermore, it uses a camera model that is capable of handling a large range of complex lens distortions that can occur in cameras that are typically used in machine vision applications. To determine the model parameters, geometrically meaningful photogrammetric error measures are used. They are independent of the parameterization of the model and typically result in a higher accuracy. We apply a stochastic model for all parameters (observations and unknowns), which allows us to assess the precision and significance of the calibrated model parameters. To evaluate our methods, we propose novel procedures that are relevant in real-world applications and do not require ground truth values. Experiments on synthetic and real data show that our approach improves the absolute positioning accuracy of industrial robots significantly. By applying our approach to two different uncalibrated UR3e robots, one guided by a camera and one by a 3D sensor, we were able to reduce the RMS evaluation error by approximately 85% for each robot.

Novel View Synthesis with Neural Radiance Fields for Industrial Robot Applications

Abstract. Neural Radiance Fields (NeRFs) have become a rapidly growing research field with the potential to revolutionize typical photogrammetric workflows, such as those used for 3D scene reconstruction. As input, NeRFs require multi-view images with corresponding camera poses as well as the interior orientation. In the typical NeRF workflow, the camera poses and the interior orientation are estimated in advance with Structure from Motion (SfM). But the quality of the resulting novel views, which depends on different parameters such as the number and distribution of available images, the accuracy of the related camera poses and interior orientation, but also the reflection characteristics of the depicted scene, is difficult to predict. In addition, SfM is a time-consuming pre-processing step, and its robustness and quality strongly depend on the image content. Furthermore, the undefined scaling factor of SfM hinders subsequent steps in which metric information is required. In this paper, we evaluate the potential of NeRFs for industrial robot applications. To start with, we propose an alternative to SfM pre-processing: we capture the input images with a calibrated camera that is attached to the end effector of an industrial robot and determine accurate camera poses with metric scale based on the robot kinematics. We then investigate the quality of the novel views by comparing them to ground truth, and by computing an internal quality measure based on ensemble methods. For evaluation purposes, we acquire multiple datasets that pose challenges for reconstruction typical of industrial applications, like reflective objects, poor texture, and fine structures. We show that the robot-based pose determination reaches similar accuracy as SfM in non-demanding cases, while having clear advantages in more challenging scenarios. We also report results of the novel view quality for different NeRF approaches, showing that an additional online pose refinement may be disadvantageous. Finally, we present first results of applying the ensemble method to estimate the quality of the synthetic novel view in the absence of a ground truth.

On-Orbit Geometric Calibration of the HJ-2 A/B Satellites' Infrared Sensors

Abstract. With the advancement of China's satellite technology, the HuanJingJianZai-2 A/B (HJ-2 A/B) satellites, equipped with whisk-broom infrared sensors, represent a significant leap forward in environmental monitoring and Earth observation capabilities. This technological leap, however, introduces new challenges in calibration. The unique structure of the HJ-2 A/B infrared spectroradiometer (IRS) necessitates innovative calibration techniques, as traditional methods primarily focused on exterior orientation parameters (EOPs) and often overlooked the importance of interior orientation accuracy, which is essential for accurate multispectral band registration and color rendering. Addressing this gap, we introduce an innovative multi-focal-plane-array joint calibration method specifically designed for whisk-broom cameras. Our method involves selecting a master band from each focal plane array for accurate focal length calibration and deriving ground control points from image matching and altitude interpolation for comprehensive bundle adjustment. This adjustment refines EOPs and interior orientation parameters (IOPs), ensuring globally optimal EOPs and enhanced IOPs calibration stability. The application of our method to the HJ-2 A/B IRS yielded substantial improvements in georeferencing and band registration accuracies, surpassing traditional methods. This paper details the multi-focal-plane-array joint calibration method, describes the IRS and experimental setup, presents the experimental results, and concludes with the implications and potential applications of our findings.

Investigation of Fiber Orientation of Fused Filament Fabricated CFRP Composites via an External Magnetic Field

Abstract For carbon fiber-reinforced plastic (CFRP) composites, controlling the interior fiber distribution and orientation during the manufacturing process is a common approach to optize the structural performance of fabricated parts. However, few studies have been conducted to investigate fiber alignment during the additive manufacturing of CFRP composites. This study proposes a magnetic field controlled (MFC) method to control the fiber orientation during the fused filament fabrication (FFF) of nickel-coated carbon fiber (NCF) reinforced polymer composites. Firstly, a theoretical analysis model is established to explore the suitable magnetic field intensity for fiber rotation. Secondly, a customized FFF system with MFC components is implemented, and a polylactic acid matrix composite containing 10 wt% NCF is printed to validate the feasibility of the proposed approach. The microstructure of the printed samples is examined to assess the effectiveness of the method. Finally, uniaxial tensile tests are performed to investigate the impact of fiber orientation adjustment on mechanical properties. The experimental results reveal that the MFC method can effectively align the interior fiber orientation of CFRP composites, leading to a significant increase in the tensile strength (approximately 8.8%) and Young's modulus (around 10.5%) of the printed samples.

Research on Improved Differential Evolution Particle Swarm Hybrid Optimization Method and Its Application in Camera Calibration

The calibration of cameras plays a critical role in close-range photogrammetry because the precision of calibration has a direct effect on the quality of results. When handling image capture using a camera, traditional swarm intelligence algorithms such as genetic algorithms and particle swarm optimization, in conjunction with Zhang’s calibration method, frequently face difficulties regarding local optima and sluggish convergence. This study presents an enhanced hybrid optimization approach utilizing both the principles of differential evolution and particle swarm optimization, which is then employed in the context of camera calibration. Initially, we establish a measurement model specific to the camera in close-range photogrammetry and determine its interior orientation parameters. Subsequently, employing these parameters as initial values, we perform global optimization and iteration using the improved hybrid optimization algorithm. The effectiveness of the proposed approach is subsequently validated through simulation and comparative experiments. Compared to alternative approaches, the proposed algorithm enhances both the accuracy of camera calibration and the convergence speed. It effectively addresses the issue of other algorithms getting trapped in local optima due to image distortion. These research findings provide theoretical support for practical engineering applications in the field of control theory and optimization to a certain extent.

V-SLAM-AIDED PHOTOGRAMMETRY TO PROCESS FISHEYE MULTI-CAMERA SYSTEMS SEQUENCES

Abstract. The advent of mobile mapping systems (MMSs) and computer vision algorithms has enriched a wide range of navigation and mapping tasks such as localisation, 3D motion estimation and 3D mapping. This study focuses on Visual Simultaneous Localisation and Mapping (V-SLAM) in the context of two in-houses MMSs: Ant3D, a patented five-fisheye multi-camera rig and GeoRizon, a high-resolution stereo fisheye rig. The aim is to leverage V-SLAM to enhance the systems performance in near-real-time and non-real-time 3D reconstruction applications. The research investigates both Monocular and Stereo V-SLAM applied to both MMSs and tackles the challenge of combining the V-SLAM estimated trajectory of one or a pair of cameras with known multi-camera relative orientation. We propose a state-of-the-art code that serves as a flexible and extensible platform for MMSs image acquisition and processing, along with an adapted version of the well-established ORB-SLAM3.0. Evaluation is performed in a cultural heritage challenging setup: the Minguzzi spiral staircase in the Duomo di Milano Cathedral. Performed tests highlight that introducing V-SLAM trajectories as well as pre-calibrated interior orientation and multi-camera constraints improve speed, applicability and accuracy of 3D surveys.

내부표정요소 편의가 Multi-light Ray Intersection에서 지상 정확도에 미치는 영향과 인자분석

내부표정요소 편의가 Multi-light Ray Intersection에서 지상 정확도에 미치는 영향과 인자분석

INTRODUCTION AND VALIDATION OF A NOVEL CALIBRATION FRAME

Abstract. The use of Photogrammetry is increasingly used by several disciplines, including marine science. Among others, the accuracy of a 3D model generated from images, depends on the quality of the calibration and stability of the camera used to capture the images. For the calibration an optimum 3D geometry is essential to minimise correlations between the camera’s interior orientation parameters. For the calibration, usually various different types of calibration frames are used. However, in practice, it can be challenging to use these frames when working in underwater environments. Calibration frames can be bulky, which makes them difficult to handle and transport, especially in boats where space is at a premium. This study aimed to develop a collapsible (and thereby portable) calibration frame, which is more practical for marine field data capture. The proposed collapsible calibration frame is validated in-air and underwater. Overall, three tests are performed. Firstly, the reliability of the frame is validated, i.e. if the collapsible frame can be put together in such a way that the Ground Control Points (GCPs) on the frame have unchanged positions relative to each other. The test showed a very small bias which could be removed by changing to a baseline assessment. Secondly, repeatability is validated, i.e. if the same results can be achieved for different software and camera combinations when using the same baselines. The test showed a clear downwards trend of the results for lower-grad cameras. However, all adjustments using the different software solutions and cameras show that the frame is suitable for application in-air. The final test is an underwater performance test which verified that the frame is usable achieving root-mean-squared error values of below 2 mm when using baselines.

Geometric Calibration of the ESA Mars Express Orbiter High Resolution Stereo Camera

AbstractThe high‐resolution stereo camera mounted on the Mars Express (MEX) spacecraft of the European Space Agency is specifically designed for planetary photogrammetry and mapping missions. High‐precision geometric calibration is required to maintain the geometric quality of satellite imaging. This study aimed to develop a Martian geometric calibration model based on the detector look angle and offset matrix to compensate for systematic errors caused by the camera system parameters, Doppler shift measurements, and other parameters of the interior and exterior orientation during positioning. Moreover, the difficulty in identifying high‐precision geometric ground control points from extra‐terrestrial imagery was addressed by proposing an iterative Mars Orbiter Laser Altimeter extraction algorithm based on the similarity measure of terrain relief. Using this algorithm, constraint conditions were established, and a calibration model was introduced to achieve in‐orbit geometric calibration of the MEX satellite. The proposed method is effective in reducing the distortion of the camera in calibration images from 12 pixels to <1 pixel, both before and after calibration. Moreover, the plane accuracy of validation images improved from approximately 900 to 200 m, while elevation accuracy improved from 500 to 100 m, verifying the effectiveness of the proposed method.

Novel Bayesian Inference-Based Approach for the Uncertainty Characterization of Zhang's Camera Calibration Method.

Camera calibration is necessary for many machine vision applications. The calibration methods are based on linear or non-linear optimization techniques that aim to find the best estimate of the camera parameters. One of the most commonly used methods in computer vision for the calibration of intrinsic camera parameters and lens distortion (interior orientation) is Zhang's method. Additionally, the uncertainty of the camera parameters is normally estimated by assuming that their variability can be explained by the images of the different poses of a checkerboard. However, the degree of reliability for both the best parameter values and their associated uncertainties has not yet been verified. Inaccurate estimates of intrinsic and extrinsic parameters during camera calibration may introduce additional biases in post-processing. This is why we propose a novel Bayesian inference-based approach that has allowed us to evaluate the degree of certainty of Zhang's camera calibration procedure. For this purpose, the a prioriprobability was assumed to be the one estimated by Zhang, and the intrinsic parameters were recalibrated by Bayesian inversion. The uncertainty of the intrinsic parameters was found to differ from the ones estimated with Zhang's method. However, the major source of inaccuracy is caused by the procedure for calculating the extrinsic parameters. The procedure used in the novel Bayesian inference-based approach significantly improves the reliability of the predictions of the image points, as it optimizes the extrinsic parameters.

Применение сферической модели камеры для решения фотограмметрических задач

In this paper, we consider the Unified Camera Model and its properties, as well as investigate and apply the improved Double-Spherical one (DS-model) in the task of restoring exterior and interior orientation elements for a set of images with ultra-wide-angle fisheye lens. The use of panoramic cameras in photogrammetry issues is a new trend, while the spherical model fully inherits the properties of the classical central-projection sample. The mathematical concept of using it is presented, as well as the solution of the optimization problem of calculating the elements of the internal and external orientation of the camera through the Gauss-Newton method, which is relevant for the tasks of reconstructing a three-dimensional scene and developing an optical SLAM based on a multi-camera system with ultra-wide-angle lenses, is implemented. In this work, epipolar geometry methods, digital image processing methods, statistical, numerical, and optimization ones were used. The results obtained enable evaluating the applicability of the DS-model for three-dimensional reconstruction of scenes during external and internal monitoring of buildings and structures, restoration of the motion trajectory and position in space for mobile systems with an installed camera, solving matters such as Perspective-n-Point for restoring the relative orientation between different active and passive sensors

First and Second-Order Polynomial Transformation for satellite image correction

The absence of ground control points (GCPs) in historical declassified intelligence satellite photography (DISP) images, as well as satellite orbit parameters, and camera interior and exterior orientation parameters, necessitated the creation of a first-order 1st polynomial equation model and a second-order 2nd polynomial equation model for orthorectification purposes. The Mosul region of northwest Iraq was used as a case study, with data from Satellite 7 released on April 15, 1999. The study achieved an RMS of approximately 2.1 x 10−8 and the error ratio was considered acceptable due to the 30-meter resolution of the satellite image.

Influence of On-Site Camera Calibration with Sub-Block of Images on the Accuracy of Spatial Data Obtained by PPK-Based UAS Photogrammetry

Unmanned Aerial Systems (UAS) Photogrammetry has become widely used for spatial data acquisition. Nowadays, RTK (Real Time Kinematic) and PPK (Post Processed Kinematic) are the main correction methods for accurate positioning used for direct measurements of camera station coordinates in UAS imagery. Thus, 3D camera coordinates are commonly used as additional observations in Bundle Block Adjustment to perform Global Navigation Satellite System-Assisted Aerial Triangulation (GNSS-AAT). This process requires accurate Interior Orientation Parameters to ensure the quality of photogrammetric intersection. Therefore, this study investigates the influence of on-site camera calibration with a sub-block of images on the accuracy of spatial data obtained by PPK-based UAS Photogrammetry. For this purpose, experiments of on-the-job camera self-calibration in the Metashape software with the SfM approach were performed. Afterward, experiments of GNSS-Assisted Aerial Triangulation with on-site calibration in the Erdas Imagine software were performed. The outcomes show that only the experiment of GNSS-AAT with three Ground Control Points yielded horizontal and vertical accuracies close to nominal precisions of the camera station positions by GNSS-PPK measurements adopted in this study, showing horizontal RMSE (Root-Mean Square Error) of 0.222 m and vertical RMSE of 0.154 m. Furthermore, the on-site camera calibration with a sub-block of images significantly improved the vertical accuracy of the spatial information extraction.

Modelling and Estimation of Interior Orientation of Non-Metric Cameras using Artificial Intelligence

With the advancement of low-cost digital cameras and their widespread, especially in smartphones, these cameras are not designed for photogrammetry applications. The main aim of this study is to model and estimate the interior orientation parameters (IOPs) for images captured by volunteer-run cameras using artificial intelligence. These cameras were unavailable to perform traditional calibration processes or use images from unknown sources for image measuring. Estimating IOPs using random values within the range determined by testing the selected sample's consistency. Optimization was performed by Utilizing the Simulated Annealing algorithm based on stereo calibration to obtain the best possible values of these parameters that produce the minimum RMS-reprojection error attained. By The variance between the parameters from the pre-calibration process and estimated by an artificial intelligence system, The coefficient of determination in the IOPs (focal length R2 = 0.717 to 0.812, principal point (X, Y) R2 = (0.674 to 0.869, 0.504 to 0.613), Both radial and tangential coefficients, R2 was close to zero). Therefore, the estimations of radial distortions k1, k2, and tangential distortions p1 and p2 are invalid. A reasonably strong relationship between principal distance and principal point with low lens distortion parameters due to the significant relative differences between the distortion parameters, sufficient strength of the relationship between calculating parameters, and estimating according to accuracy tolerance in photogrammetry applications.

Understanding microtopography changes in agricultural landscapes through precision assessments of digital surface models by the UAV-RTK-PPK method without ground control points

Soil erosion on agricultural land causes severe environmental problems and damages crop plants. Structure-from-motion with multiview stereo (SfM-MVS) together with data obtained via unmanned aerial vehicles (UAVs) helps understand spatiotemporal changes in the ground surface if there is enough precise data. However, installing ground control points in the field is labor intensive and disturbs field conditions. Here, images georeferenced using the Real-Time Kinematic (RTK) method were further improved using Post-Processing Kinematic (PPK) analysis. This study aims to verify whether the UAV-RTK-PPK method can create precise digital surface models to evaluate topographic changes owing to erosion without ground control points. The effects of the camera's interior orientation parameters settings and the addition of oblique images to nadiral images on the precision were also examined to reduce the positional error by the doming and bowling phenomena. Field observations were conducted in a 150 m × 25 m potato field with 156 poles to consider the three-dimensional object shapes in the verification. The doming or bowling phenomena occurred using only the nadiral images. The precision of the digital surface model was approximately 0.40 and 0.20 m (in the Z-direction) using the estimated and fixed camera's interior orientation parameters, respectively. In contrast, the digital surface models produced by adding oblique images to the nadiral images had a precision of approximately 0.04 m, regardless of the camera's interior orientation parameters. Thus, a high-precision digital surface model was obtained from images acquired using the UAV-RTK-PPK and SfM-MVS methods without ground control points. The digital surface model positional precision was improved by adding oblique images to the nadiral images without having to consider the camera's interior orientation parameters. Topographic changes >0.05 m after erosion by heavy rainfall were evaluated without ground control points using the UAV-RTK-PPK method with oblique and nadiral images. Therefore, our method can monitor the erosion of agricultural fields that may cause crop damage, such as the greening of potatoes.

Giving Historical Photographs a New Perspective: Introducing Camera Orientation Parameters as New Metadata in a Large-Scale 4D Application

The ongoing digitization of historical photographs in archives allows investigating the quality, quantity, and distribution of these images. However, the exact interior and exterior camera orientations of these photographs are usually lost during the digitization process. The proposed method uses content-based image retrieval (CBIR) to filter exterior images of single buildings in combination with metadata information. The retrieved photographs are automatically processed in an adapted structure-from-motion (SfM) pipeline to determine the camera parameters. In an interactive georeferencing process, the calculated camera positions are transferred into a global coordinate system. As all image and camera data are efficiently stored in the proposed 4D database, they can be conveniently accessed afterward to georeference newly digitized images by using photogrammetric triangulation and spatial resection. The results show that the CBIR and the subsequent SfM are robust methods for various kinds of buildings and different quantity of data. The absolute accuracy of the camera positions after georeferencing lies in the range of a few meters likely introduced by the inaccurate LOD2 models used for transformation. The proposed photogrammetric method, the database structure, and the 4D visualization interface enable adding historical urban photographs and 3D models from other locations.

A Combined Physical and Mathematical Calibration Method for Low-Cost Cameras in the Air and Underwater Environment.

Low-cost camera calibration is vital in air and underwater photogrammetric applications. However, various lens distortions and the underwater environment influence are difficult to be covered by a universal distortion compensation model, and the residual distortions may still remain after conventional calibration. In this paper, we propose a combined physical and mathematical camera calibration method for low-cost cameras, which can adapt to both in-air and underwater environments. The commonly used physical distortion models are integrated to describe the image distortions. The combination is a high-order polynomial, which can be considered as basis functions to successively approximate the image deformation from the point of view of mathematical approximation. The calibration process is repeated until certain criteria are met and the distortions are reduced to a minimum. At the end, several sets of distortion parameters and stable camera interior orientation (IO) parameters act as the final camera calibration results. The Canon and GoPro in-air calibration experiments show that GoPro owns distortions seven times larger than Canon. Most Canon distortions have been described with the Australis model, while most decentering distortions for GoPro still exist. Using the proposed method, all the Canon and GoPro distortions are decreased to close to 0 after four calibrations. Meanwhile, the stable camera IO parameters are obtained. The GoPro Hero 5 Black underwater calibration indicates that four sets of distortion parameters and stable camera IO parameters are obtained after four calibrations. The camera calibration results show a difference between the underwater environment and air owing to the refractive and asymmetric environment effects. In summary, the proposed method improves the accuracy compared with the conventional method, which could be a flexible way to calibrate low-cost cameras for high accurate in-air and underwater measurement and 3D modeling applications.

Optical design and precision analysis of a single-line-array pendulum sweep high-resolution mapping camera.

With the improvement of the satellite resolution, it is urgent to develop the single-line-array mapping camera. However, the camera accuracy is influenced by the satellite attitude's rapid maneuvering during the imaging process. In our study, a coaxial four-mirror optical system with a field bias with a focal length of 7050mm, F-number of 10.8, field of view of 1.2°, and spectral range of 450-800nm is designed. By combining mathematical modeling and ray tracing, the offset of the camera interior orientation elements caused by the misalignment of the secondary mirror is derived. The simulation results show that the maximum relative error does not exceed 2.119%. Besides, a desensitization design method based on the magnification parameter control method is proposed, and the results show that the sensitivity of camera interior orientation elements to the secondary mirror is reduced, indicating the effectiveness of the system desensitization design, which is of great significance for the improvement of camera accuracy.

WINTER THERMAL COMFORT OF A TYPICAL COURTYARD GEOMETRY IN A SEMI-ARID CLIMATE

ABSTRACTThe courtyard is an ancient outdoor design space surrounded by walls or buildings, acting as a microclimate modifier in hot-dry climates. This is related to its geometry, such as high proportions of height-to-width (H/W) and north-south (NS) orientation by providing shade and decreasing heat stress on hot summer days. However, its effect during winter still needs to be discussed, especially in hot summer and cold winter conditions (such as in semi-arid climates). This research studies the winter thermal comfort of a typical courtyard geometry suitable for hot summer conditions in a semi-arid climate. A literature review supports the identification of typical courtyard geometry addressed for hot summer conditions. Then, field measurements of the external surface temperature of courtyard interior orientations, microclimatic variables, and the Predicted Mean Votes (PMV) of occupants inside the courtyard were performed. The results indicate high correlations between PMV and courtyard interior orientations with the H/W ratio regarding cold stress. Thus, north-east and South-East orientations and H/W ratio less than (< 0.8) are recommended for better winter environmental conditions in semi-arid climates.

ПРОВЕДЕНИЕ КАЛИБРОВКИ НЕМЕТРИЧЕСКОЙ ФОТОКАМЕРЫ В БЕСПИЛОТНОМ ЛЕТАТЕЛЬНОМ АППАРАТЕ ПРИ МОНИТОРИНГЕ ЗЕМЕЛЬ

Recently, unmanned aerial vehicles (UAVs) have been widely used in our country and in the world. This is due to continuity, endurance, the ability to work under overload conditions and the exclusion of the human factor. In recent years, there has been a revolutionary leap in the development of UAVs, which made it possible to divide this market segment into completely different price categories. The increase in the use of UAVs has made this product really in demand and was able to introduce it into various areas of life. This also applies to geodetic measurements. Photogrammetry methods began to work closely with methods of easy-to-control unmanned aerial vehicles. But at the same time, the demand for filming equipment is growing, the pricing policy of which sometimes considerably exceeds the cost of the aircraft itself, which complicates photogrammetric work. This aspect has led to research on the possibility of using inexpensive non-professional cameras on UAVs for photogrammetry, due to obtaining accurate measurements. For such cameras, there is such a thing as calibration, which includes the definition of interior orientation elements. This article discusses the use of non-metric cameras on UAVs in order to reduce the cost of work to monitor agricultural land. The following materials were used as initial data in this work: test images from the UAV at the test site; software such as SAS.Planet, MatLAb, MdCockpit V2.6.2.6, PHOTOMOD. To perform this work, the following equipment was used: unmanned aerial vehicle; non-metric digital camera. To calibrate the camera, the terrain was surveyed using Zala 421-21.