Camera Calibration Research Articles

Research on the application of virtual reality technology in contemporary environmental design

Abstract The development and application of virtual reality technology can help the innovative application of environmental design. This paper first focuses on the application of virtual reality technology in the natural environment based on the development of environmental design and the integration of natural geographical conditions. In fact, the focus is on the analysis of virtual reality technology. Selected 3D reconstruction technology algorithms, including camera calibration profile, pair of polar geometry and single response matrix, matching cost and quality assessment index. Finally, the natural environment simulation experiment is constructed based on virtual reality technology. The average real-time rendering FPS of the unoptimized vegetation distribution interval decreases continuously with the increase in the number of vegetation models. When the average vegetation number is around 54,672, the average FPS decreases to around per 53. When the average vegetation count is at 97,713, the average FPS drops to about 28 frames per second. The average real-time dye FPS stabilizes at around 50 after using the LOD multi-detail hierarchical model.

SurgT challenge: Benchmark of soft-tissue trackers for robotic surgery

This paper introduces the “SurgT: Surgical Tracking” challenge which was organized in conjunction with the 25th International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI 2022). There were two purposes for the creation of this challenge: (1) the establishment of the first standardized benchmark for the research community to assess soft-tissue trackers; and (2) to encourage the development of unsupervised deep learning methods, given the lack of annotated data in surgery. A dataset of 157 stereo endoscopic videos from 20 clinical cases, along with stereo camera calibration parameters, have been provided. Participants were assigned the task of developing algorithms to track the movement of soft tissues, represented by bounding boxes, in stereo endoscopic videos. At the end of the challenge, the developed methods were assessed on a previously hidden test subset. This assessment uses benchmarking metrics that were purposely developed for this challenge, to verify the efficacy of unsupervised deep learning algorithms in tracking soft-tissue. The metric used for ranking the methods was the Expected Average Overlap (EAO) score, which measures the average overlap between a tracker’s and the ground truth bounding boxes. Coming first in the challenge was the deep learning submission by ICVS-2Ai with a superior EAO score of 0.617. This method employs ARFlow to estimate unsupervised dense optical flow from cropped images, using photometric and regularization losses. Second, Jmees with an EAO of 0.583, uses deep learning for surgical tool segmentation on top of a non-deep learning baseline method: CSRT. CSRT by itself scores a similar EAO of 0.563. The results from this challenge show that currently, non-deep learning methods are still competitive. The dataset and benchmarking tool created for this challenge have been made publicly available at https://surgt.grand-challenge.org/. This challenge is expected to contribute to the development of autonomous robotic surgery and other digital surgical technologies.

A novel camera calibration method based on circle projection model

Camera calibration is one of the important factors affecting measurement accuracy in the field of visual measurement, and the camera calibration method with high precision and strong robustness is the prerequisite for ensuring high-precision visual measurement. Among the commonly used calibration methods, target-based calibration methods have high accuracy and good flexibility. However, the feature extraction of checkerboard targets is prone to noise interference. Circle targets have higher feature extraction accuracy, but will introduce eccentricity errors in the center of the circle projection. In order to eliminate the eccentricity error of circle targets, this paper proposes a camera calibration method based on the circle projection model, which does not rely on the central projection coordinates, but directly constructs the mapping between the circle equation on the target and the elliptic equation in the camera image, and solves the optimal mapping parameters through nonlinear optimization algorithm. Through simulation and experiments, it has been proven that the calibration method based on the circle projection model has high calibration accuracy and anti-noise interference ability, providing a guarantee for high-precision visual measurement.

The impact of oblique images and flight‐planning scenarios on the accuracy of UAV 3D mapping

AbstractThe developments in lightweight unmanned aerial vehicles (UAVs) and structure‐from‐motion (SfM)‐based software have opened a new era in 3D mapping which is notably cost‐effective and fast, though the photogrammetric blocks lead to systematic height error due to inaccurate camera calibration parameters particularly when the ground control points (GCPs) are few and unevenly distributed. The use of onboard Global Navigation Satellite System (GNSS) receivers (such as RTK‐ or PPK‐based devices which use the DGNSS technique) to obtain the accurate coordinates of camera perspective centres has reduced the need for ground surveys, nevertheless, the aforementioned systematic error was reported in the UAV photogrammetric blocks. In this research, three flight‐planning scenarios with oblique imagery in addition to the traditional nadir block were evaluated and processed with four different processing cases. Therefore, 16 various blocks with different overlaps, direct and indirect georeferencing approaches as well as flight‐planning scenarios were tested to examine and offer the best imaging network. The results denote that the combination of oblique images located on a circle in the centre of the block with the nadir block provides the best self‐calibration functionality and improves the final accuracy by 50% (from 0.163 to 0.085 m) for direct georeferenced blocks and by 40% (from 0.042 to 0.026 m) for indirect georeferenced blocks.

Checkerboard corner detection method based onneighborhood linear fitting.

To improve the calibration accuracy of a vision measurement system, a checkerboard corner detection method based on linear fitting of the checkerboard local contour is proposed. First, by binarization and morphological dilation of the checkerboard image, the coordinates of two adjacent vertices of adjacent dark squares are obtained; the midpoint of the two vertices is taken as the reference point; the reference dotted array is obtained; and the Zernike moment subpixel method is used to obtain the checkerboard contour data points in the neighborhood of each reference point. Finally, the contour points are classified according to the orientation based on the reference points; two intersecting lines are fitted; and the intersection of the two lines is exactly the corner point that we want to find. A camera calibration experiment was conducted on the same group of checkerboard images. The results show that the calibration results of the corner points obtained based on this method are highly consistent with the OpenCV library function method and the MATLAB Toolbox calibration method, and the reprojection error is smaller. At the same time, it is robust to changes in the light source brightness.

A Systematic Stereo Camera Calibration Strategy: Leveraging Latin Hypercube Sampling and 2k Full-Factorial Design of Experiment Methods.

This research aimed to optimize the camera calibration process by identifying the optimal distance and angle for capturing checkered board images, with a specific focus on understanding the factors that influence the reprojection error (ϵRP). The objective was to improve calibration efficiency by exploring the impacts of distance and orientation factors and the feasibility of independently manipulating these factors. The study employed Zhang's camera calibration method, along with the 2k full-factorial analysis method and the Latin Hypercube Sampling (LHS) method, to identify the optimal calibration parameters. Three calibration methods were devised: calibration with distance factors (D, H, V), orientation factors (R, P, Y), and the combined two influential factors from both sets of factors. The calibration study was carried out with three different stereo cameras. The results indicate that D is the most influential factor, while H and V are nearly equally influential for method A; P and R are the two most influential orientation factors for method B. Compared to Zhang's method alone, on average, methods A, B, and C reduce ϵRP by 25%, 24%, and 34%, respectively. However, method C requires about 10% more calibration images than methods A and B combined. For applications where lower value of ϵRP is required, method C is recommended. This study provides valuable insights into the factors affecting ϵRP in calibration processes. The proposed methods can be used to improve the calibration accuracy for stereo cameras for the applications in object detection and ranging. The findings expand our understanding of camera calibration, particularly the influence of distance and orientation factors, making significant contributions to camera calibration procedures.

A Binocular Camera Calibration Method in Froth Flotation Based on Key Frame Sequences and Weighted Normalized Tilt Difference

3D froth information is a significant indicator of working condition recognition in froth flotation. To extract it accurately, the first and key step is camera calibration. Although there have been many camera calibration methods proposed in these years, they cannot handle well for binocular camera calibration in froth flotation. The main reason is that there is an open environment without support surface or special geometric shapes for camera calibration in froth flotation. Therefore, we propose a novel binocular camera calibration method based on key frame sequences and weighted normalized tilt difference. First, we extract a high-quality frame sequence by filtering out incomplete and out-of-focus images from the froth video. Then, we propose a method to measure the tilt degree of the planar plate and extract the key frame sequences of the planar plate based on the tilt difference. After that, we calculate the camera parameters by weighted normalized tilt difference under various poses. The proposed binocular camera calibration method uses the froth video to obtain accurate and robust camera parameters, and does not require auxiliary equipment. Experiments have demonstrated the effectiveness and flexibility of the proposed method in froth flotation.

MVP-Human Dataset for 3-D Clothed Human Avatar Reconstruction From Multiple Frames

In this paper, we consider a novel problem of reconstructing a 3D clothed human avatar from multiple frames, independent of assumptions on camera calibration, capture space, and constrained actions. We contribute a large-scale dataset, Multi-View and multi-Pose 3D human (MVP-Human in short) to help address this problem. The dataset contains 400 subjects, each of which has 15 scans in different poses and 8-view images for each pose, providing 6,000 3D scans and 48,000 images in total. In addition, a baseline method that takes multiple images as inputs, and generates a shape-with-skinning avatar in the canonical space, finished in one feed-forward pass is proposed. It first reconstructs the implicit skinning fields in a multi-level manner, and then the image features from multiple images are aligned and integrated to estimate a pixel-aligned implicit function that represents the clothed shape. With the newly collected dataset and the baseline method, it shows promising performance on 3D clothed avatar reconstruction. We release the MVP-Human dataset and the baseline method in, hoping to promote research and development in this field.

SVBRDF Reconstruction by Transferring Lighting Knowledge

AbstractThe problem of reconstructing spatially‐varying BRDFs from RGB images has been studied for decades. Researchers found themselves in a dilemma: opting for either higher quality with the inconvenience of camera and light calibration, or greater convenience at the expense of compromised quality without complex setups. We address this challenge by introducing a two‐branch network to learn the lighting effects in images. The two branches, referred to as Light‐known and Light‐aware, diverge in their need for light information. The Light‐aware branch is guided by the Light‐known branch to acquire the knowledge of discerning light effects and surface reflectance properties, but without the reliance of light positions. Both branches are trained using the synthetic dataset, but during testing on real‐world cases without calibration, only the Light‐aware branch is activated. To facilitate a more effective utilization of various light conditions, we employ gated recurrent units (GRUs) to fuse the features extracted from different images. The two modules mutually benefit when multiple inputs are provided. We present our reconstructed results on both synthetic and real‐world examples, demonstrating high quality while maintaining a lightweight characteristic in comparison to previous methods.

Four-dimensional particle tracking velocimetry measurements of unsteady three-dimensional vortex onset and progression for 5415 straight ahead, static drift, and pure sway

Large-scale towed system four-dimensional particle tracking velocimetry (4DPTV) implementation and measurements are presented for towing tank tests for a naval combatant 5415 ship model for straight ahead, static drift β = 10°, and pure sway βmax = 10° conditions. The results, for the first time, provide instantaneous volumetric flow field data around the ship model including near the hull surface, and a complete description of the sonar dome 3D unsteady vortex separation onset and progression due to the 4DPTV larger measurement volume and higher data rates in comparison with previous results using tomographic particle image velocimetry (TPIV); thereby, providing the essential data required for the assessment of current hybrid Reynolds-averaged Navier–Stokes/large eddy simulation turbulence modeling capabilities and guidance for the necessary modeling improvements. The 4DPTV system is summarized and compared with the previous TPIV, including the camera calibration procedures, trigger systems, and synchronization with the sway motion. Analysis procedures, including data reduction procedures and the 3D vortex core detection and voxel labeling techniques, are described. The identification of the complex vortex separations and breakdown is aided using complementary detached eddy simulations. The vortex–vortex interaction process is visualized from instantaneous volumetric datasets. The pros and cons of the 4DPTV vs TPIV, the comparison between the second- and third-generation vortex visualization technique, and the statistical convergence error analysis of sonar dome port vortex for β = 10° are discussed. Plans for increased spatial resolution of the 4DPTV system and additional data reduction techniques for detailed turbulence analysis are also discussed.

Deep learning-based binocular composite color fringe projection profilometry for fast 3D measurements

At present, deep learning plays a crucial role in structured light 3D reconstruction. Further, in the field of fringe projection profilometry, learning 3D features from fringes and performing 3D reconstruction are being studied by many researchers. This paper combines deep learning with binocular fringe projection, uses three channels to form a single-composite-color fringe pattern for three different frequencies as the input, and predicts the numerator and denominator required to solve the wrapped phase of the object. The wrapped phase is calculated using an arctangent function. The absolute phase is obtained by unwrapping the multi-frequency heterodyne method, and the absolute phase of the left and right cameras is matched to obtain a disparity map. The parameters obtained through camera calibration can restore the 3D shape of the object, which greatly reduces the number of fringes required; accuracy close to the phase of the training set is achieved. Finally, the experimental results demonstrate the feasibility of this approach.

Hardware and Software Implementation of the Embedded Controlling System for the TuMag Camera

The main objective of this paper is to present the design and implementation of an embedded controlling system for the Tunable Magnetograph (TuMag) instrument camera sensor and test-bench software application for camera laboratory characterization. The TuMag camera is based on a new scientific sensor GPIXEL SENSE400 device family equipped with an FPGA. The camera sensor has an active area of 2048 × 2048 pixels and reaches a frame rate of up to 48 frames per second. The embedded controlling system was implemented on an Artix 7 FPGA, which oversees controlling the image sensor through a configuration interface. It controls the read-out of the sensor data and the communication of commands with the host device. The camera has a standard CoaXPress communication interface of up to 3125 Gbps. The FPGA embedded control system allows bit, word, and serial channel calibration and the modification of several parameters, such as the region of interest (RoI) size, exposition time, hardware or software trigger, single or continuous acquisition modes, sensor gain, and adjustment of the black level offset and exposition time. In addition, the firmware has a temperature controller, voltage level monitoring, and an enable–disable power sensor. In addition, the test-bench software application is a library developed in Python 3.7. It is a wrapper for the standard GeniCan commercial frame grabber interpreter. This wrapper permits the improvement of the interaction and interface between the user and the hardware in the camera calibration. The test-bench software application permits the reduction of costs and the transport risk of the TuMag camera between different laboratories. All TuMag embedded control systems and test-bench software application functionalities were successfully tested according to scientific requirements.

Effective camera calibration by using phase-shifting fringe patterns

The accuracy of conventional target-based camera calibration depends on the detection of feature points. To improve the calibration accuracy in the flat field, this paper proposes an innovative calibration method based on encoded phase-shifting fringe (PSF) patterns. By using arbitrary quadrilateral interpolation, the relationship between a point on the virtual phase plane and the corresponding four original phase points is determined, establishing a mapping between the simplified original phase field and the virtual phase field. This spatial relationship can be used for accurate camera calibration. The proposed research investigates the details of the PSF based camera calibration method and demonstrates its accuracy and effectiveness through experiments. The evaluation of this method includes the checkerboard edge length variance analysis and the linear distance analysis of the checkerboard fit, both of which serve as validation metrics. Camera distortion is described by the phase distribution, and the average phase error after correction is less than 0.001 pixels (px).

Stereovision-based vibration measurement of stay cable using synchronized multi-camera setup and video motion magnification

This paper proposes a new framework for 3D noncontact stereovision-based displacement measurement in conjunction with Phase-based Video Motion Magnification (PVMM) to capture in-plane and out-of-plane vibrations of stay cables. In the proposed method, cable vibrations are captured by a pair of synchronized digital cameras, then PVMM method is applied to magnify micro vibrations of the cable (ambient responses) recorded by video cameras. Next, a camera calibration technique is proposed to convert the 2D image coordinates to 3D global coordinates, followed by the stereo calibration between the two cameras to synchronize the target points. Afterwards, a Centroid-based Bounding-Box Tracking (CBBT) technique is proposed and used to track the movements of the cable in each video frame. As a result, time-history displacements of the cable are extracted from the magnified videos. Finally, natural frequencies and damping ratios of the cable are extracted from the in-plane and out-of-plane displacements using Wavelet-screening Random Decrement Technique (WS-RDT). The proposed methodology has been validated through a couple of wind-tunnel tests of cable vibrations. The results have demonstrated a good agreement between the proposed method and conventional accelerometer-based measurement results.

Accurate Visual Simultaneous Localization and Mapping (SLAM) against Around View Monitor (AVM) Distortion Error Using Weighted Generalized Iterative Closest Point (GICP).

Accurately estimating the pose of a vehicle is important for autonomous parking. The study of around view monitor (AVM)-based visual Simultaneous Localization and Mapping (SLAM) has gained attention due to its affordability, commercial availability, and suitability for parking scenarios characterized by rapid rotations and back-and-forth movements of the vehicle. In real-world environments, however, the performance of AVM-based visual SLAM is degraded by AVM distortion errors resulting from an inaccurate camera calibration. Therefore, this paper presents an AVM-based visual SLAM for autonomous parking which is robust against AVM distortion errors. A deep learning network is employed to assign weights to parking line features based on the extent of the AVM distortion error. To obtain training data while minimizing human effort, three-dimensional (3D) Light Detection and Ranging (LiDAR) data and official parking lot guidelines are utilized. The output of the trained network model is incorporated into weighted Generalized Iterative Closest Point (GICP) for vehicle localization under distortion error conditions. The experimental results demonstrate that the proposed method reduces localization errors by an average of 39% compared with previous AVM-based visual SLAM approaches.

Novel Bayesian Inference-Based Approach for the Uncertainty Characterization of Zhang's Camera Calibration Method.

Camera calibration is necessary for many machine vision applications. The calibration methods are based on linear or non-linear optimization techniques that aim to find the best estimate of the camera parameters. One of the most commonly used methods in computer vision for the calibration of intrinsic camera parameters and lens distortion (interior orientation) is Zhang's method. Additionally, the uncertainty of the camera parameters is normally estimated by assuming that their variability can be explained by the images of the different poses of a checkerboard. However, the degree of reliability for both the best parameter values and their associated uncertainties has not yet been verified. Inaccurate estimates of intrinsic and extrinsic parameters during camera calibration may introduce additional biases in post-processing. This is why we propose a novel Bayesian inference-based approach that has allowed us to evaluate the degree of certainty of Zhang's camera calibration procedure. For this purpose, the a prioriprobability was assumed to be the one estimated by Zhang, and the intrinsic parameters were recalibrated by Bayesian inversion. The uncertainty of the intrinsic parameters was found to differ from the ones estimated with Zhang's method. However, the major source of inaccuracy is caused by the procedure for calculating the extrinsic parameters. The procedure used in the novel Bayesian inference-based approach significantly improves the reliability of the predictions of the image points, as it optimizes the extrinsic parameters.

Uncrewed aerial vehicle radiometric calibration: A comparison of autoexposure and fixed‐exposure images

AbstractRemote sensing with uncrewed aerial vehicles (UAVs) is increasingly being used in agriculture to provide data on the physical characteristics of plants under field conditions. Data accuracy is critical for decision making with a high degree of confidence. In this work, we compared two multispectral camera calibration methods for image data collected with a UAV: (1) an autoexposure method that relies on a single calibration panel and a post hoc calibration, and (2) a fixed‐exposure system that uses three in‐field gray calibration panels using the empirical line calibration method. Both methods were compared to reflectance data from (a) four ground calibration targets measured with a spectroradiometer and (b) a single manned aircraft image calibrated with commercial calibration tarps. In a band‐by‐band comparison, the autoexposure method produced almost twice as much radiometric error on average compared with fixed exposure. Because remote sensing data are commonly converted to spectral indices, the calibration methods were also evaluated by calculating the visible atmospherically resistant index (VARI) and comparing the resulting data to the manned aircraft image. Similarly, the autoexposure method in this case produced twice the error of the fixed‐exposure method. The effect of the error was considered in a production agriculture context by simulating a remote sensing‐based prescription map for pesticide application in a cotton (Gossypium) field and calculating the number of mislabeled management zones. The simulation showed that the autoexposure method would be more costly to the farm because of its higher error, roughly $8.00/ha based on the assumptions made.

An improved camera model for oblique-viewing laparoscopes: high reprojection accuracy independent of telescope rotation

Objective. Oblique-viewing laparoscopes are popular in laparoscopic surgeries where the target anatomy is located in narrow areas. Their viewing direction can be shifted by telescope rotation without changing the laparoscope pose. This rotation also changes laparoscope camera parameters that are estimated by camera calibration to be able to reproject an anatomical model onto the laparoscopic view, creating augmented reality (AR). The aim of this study was to develop a camera model that accounts for these changes, achieving high reprojection accuracy for any telescope rotation. Approach. Camera parameters were acquired by calibrations encompassing a wide telescope rotation range. For those parameters showing periodic changes upon rotation, interpolation models were created and used to establish an updatable camera model. With this model, corner points of a tracked checkerboard were reprojected onto the checkerboard laparoscopic images, at random rotation angles. Root-mean-square reprojection errors (RMSEs) were calculated between the reprojected and imaged corner points. Main results. Reprojection RMSEs were low and approximately independent on telescope rotation angle, over a wide rotation range of 320°. The mean reprojection RMSE was 2.8 0.7 pixels for a conventional laparoscope and 3.6 0.7 pixels for a chip-on-the-tip (COTT) laparoscope, corresponding to 0.3 0.1 mm and 0.4 0.1 mm in world coordinates respectively. Worst-case reprojection errors were about 9 pixels (0.8 mm) for both laparoscopes. Significance. The camera model developed in this study improves on existing models for oblique-viewing laparoscopes because it provides high reprojection accuracy independent of the telescope rotation angle and is applicable for conventional and chip-on-a-tip oblique-viewing laparoscopes. The work presented here is an important step towards creating accurate AR in image-guided interventions where oblique-viewing laparoscopes are used while simultaneously providing the surgeon the flexibility to rotate the telescope to any desired rotation angle. Acronyms. CC: camera coordinates; CCToolbox: camera calibration toolbox; COTT: chip-on-the-tip; CS: camera sensor; DD: decentering distortion; FL: focal length; OTS: optical tracking system; PP: principal point; RD: radial distortion; SI: supplementary information; hand-eye translation component

IMPROVING CAMERA POSE ESTIMATION USING SWARM PARTICLE ALGORITHMS

Abstract. Most computer vision and photogrammetry applications rely on accurately estimating the camera pose, such as visual navigation, motion tracking, stereo photogrammetry, and structure from motion. The Essential matrix is a well-known model in computer vision that provides information about the relative orientation between two images, including the rotation and translation, for calibrated cameras with a known camera matrix. To estimate the Essential matrix, the camera calibration matrices, which include focal length and principal point location must be known, and the estimation process typically requires at least five matching points and the use of robust algorithms, such as RANSAC to fit a model to the data as a robust estimator. From the usually large number of matched points, choosing five points, the Essential matrix can be determined based on a simple solution, which could be good or bad. Obtaining a globally optimal and accurate camera pose estimation, however, requires additional steps, such as using evolutionary algorithms (EA) or swarm algorithms (SA), to prevent getting trapped in local optima by searching for solutions within a potentially huge solution space.This paper aims to introduce an improved method for estimating the Essential matrix using swarm particle algorithms that are known to efficiently solve complex problems. Various optimization techniques, including EAs and SAs, such as Particle Swarm Optimization (PSO), Gray Wolf Optimization (GWO), Improved Gray Wolf Optimization (IGWO), Genetic Algorithm (GA), Salp Swarm Algorithm (SSA) and Whale Optimization Algorithm (WOA), are explored to obtain the global minimum of the reprojection error for the five-point Essential matrix estimation based on using symmetric geometric error cost function. The experimental results on a dataset with known camera orientation demonstrate that the IGWO method has achieved the best score compared to other techniques and significantly speeds up the camera pose estimation for larger number of point pairs in contrast to traditional methods that use the collinearity equations in an iterative adjustment.

A Perceptual Measure for Deep Single Image Camera and Lens Calibration.

Image editing and compositing have become ubiquitous in entertainment, from digital art to AR and VR experiences. To produce beautiful composites, the camera needs to be geometrically calibrated, which can be tedious and requires a physical calibration target. In place of the traditional multi-image calibration process, we propose to infer the camera calibration parameters such as pitch, roll, field of view, and lens distortion directly from a single image using a deep convolutional neural network. We train this network using automatically generated samples from a large-scale panorama dataset, yielding competitive accuracy in terms of standard l2 error. However, we argue that minimizing such standard error metrics might not be optimal for many applications. In this work, we investigate human sensitivity to inaccuracies in geometric camera calibration. To this end, we conduct a large-scale human perception study where we ask participants to judge the realism of 3D objects composited with correct and biased camera calibration parameters. Based on this study, we develop a new perceptual measure for camera calibration and demonstrate that our deep calibration network outperforms previous single-image based calibration methods both on standard metrics as well as on this novel perceptual measure. Finally, we demonstrate the use of our calibration network for several applications, including virtual object insertion, image retrieval, and compositing.