Camera Parameters Research Articles

Seamless and Aligned Texture Optimization for 3D Reconstruction

AbstractRestoring the appearance of the model is a crucial step for achieving realistic 3D reconstruction. High‐fidelity textures can also conceal some geometric defects. Since the estimated camera parameters and reconstructed geometry usually contain errors, subsequent texture mapping often suffers from undesirable visual artifacts such as blurring, ghosting, and visual seams. In particular, significant misalignment between the reconstructed model and the registered images will lead to texturing the mesh with inconsistent image regions. However, eliminating various artifacts to generate high‐quality textures remains a challenge. In this paper, we address this issue by designing a texture optimization method to generate seamless and aligned textures for 3D reconstruction. The main idea is to detect misalignment regions between images and geometry and exclude them from texture mapping. To handle the texture holes caused by these excluded regions, a cross‐patch texture hole‐filling method is proposed, which can also synthesize plausible textures for invisible faces. Moreover, for better stitching of the textures from different views, an improved camera pose optimization is present by introducing color adjustment and boundary point sampling. Experimental results show that the proposed method can eliminate the artifacts caused by inaccurate input data robustly and produce high‐quality texture results compared with state‐of‐the‐art methods.

Online extrinsic parameters calibration of on-board stereo cameras based on certifiable optimization

The extrinsic parameters of on-board stereo cameras can be slightly altered due to temperature fluctuations, vibrations, and accidental impacts during automobile driving, leading to significant performance loss in dense stereo matching. In this paper, we propose an online calibration method based on certifiable optimization to address this issue. Initially, sparse feature points are collected based on plane distribution and disparity. A robust optimization model is then developed to minimize the epipolar error, utilizing iterative local optimization to eliminate outliers and determine the 5DOF extrinsic parameters. Subsequently, a relaxation problem is constructed using inliers, and global optimization is performed to certify that the locally optimal results are indeed globally optimal. Comparative experimental results demonstrate that the proposed method offers high accuracy and reliability. Additionally, the quality of the disparity map generated by our calibration method is comparable to that achieved through offline calibration.

Real-time robust liver and gallbladder segmentation during laparoscopic cholecystectomy using convolutional neural networks: an analysis

Aim: Images in different laparoscopic cholecystectomy datasets are acquired using various camera models, parameters, and settings, with the annotation methods varying by institution. These factors result in inconsistent inference performance of the network model. This study aims to identify the optimal network model architecture for liver and gallbladder segmentation from several options. Then, the performance and robustness of the optimal network model are evaluated using an independent dataset that is not included in the training. Methods: The public dataset, CholecSeg8k, was utilized as the input for the network model training, validation, and testing. A local private dataset from KPJ Damansara Hospital, Selangor, Malaysia, was used for testing purposes only. For the implementation of liver and gallbladder segmentation, segmentation models, a public Python library was employed. Results: Among the experiments, highly accurate liver and gallbladder segmentation results were achieved using the feature pyramid network (FPN) architecture as the network model, with the Inception-ResNet-v2 architecture as the network backbone. The best-trained network model resulted in a loss of 0.070955, a mean intersection over union (IoU) score of 0.95896, and a mean F1-score of 0.9773 on the test set. However, visualized results for the private dataset contained considerable false-negative areas. Conclusion: The proposed automated technique has the potential to serve as an alternative to the conventional indocyanine green injection along with near-infrared fluorescence imaging (ICG-NIRF)-based method for liver and gallbladder segmentation during laparoscopic cholecystectomy. Future work will focus on enhancing the results of the private dataset. Additionally, a surgeon-assistant robotic arm that will use the liver and gallbladder segmentation results for camera steering will be analyzed.

Pupil-centric imaging model for zoom camera calibration.

Zoom camera calibration has always been challenging, as arbitrary zoom/focus settings change the camera parameters. Current calibration methods are based on the pinhole imaging model, which results in coupled calibration parameters due to the lack of geometric/physical constraints. In this Letter, we present a novel, to the best of our knowledge, pupil-centric imaging model that accounts for the camera's radial, decentering, and mustache distortions at various zoom settings using exit pupil offsets and a non-frontal lens-sensor model. Therefore, we provide a reasonable physical explanation for the different distortion effects. Global optimization is performed based on the proposed initial camera calibration and bundle adjustment under several zoom and autofocus setting combinations. Experiments using three representative zoom cameras demonstrate the effectiveness of the proposed method. Its relative measurement accuracy is better than that of current state-of-the-art methods.

NeRFtrinsic Four: An end-to-end trainable NeRF jointly optimizing diverse intrinsic and extrinsic camera parameters

Novel view synthesis using neural radiance fields (NeRF) is the state-of-the-art technique for generating high-quality images from novel viewpoints. Existing methods require a priori knowledge about extrinsic and intrinsic camera parameters. This limits their applicability to synthetic scenes, or real-world scenarios with the necessity of a preprocessing step. Current research on the joint optimization of camera parameters and NeRF focuses on refining noisy extrinsic camera parameters and often relies on the preprocessing of intrinsic camera parameters. Further approaches are limited to cover only one single camera intrinsic. To address these limitations, we propose a novel end-to-end trainable approach called NeRFtrinsic Four. We utilize Gaussian Fourier features to estimate extrinsic camera parameters and dynamically predict varying intrinsic camera parameters through the supervision of the projection error. Our approach outperforms existing joint optimization methods on LLFF and BLEFF. In addition to these existing datasets, we introduce a new dataset called iFF with varying intrinsic camera parameters. NeRFtrinsic Four is a step forward in joint optimization NeRF-based view synthesis and enables more realistic and flexible rendering in real-world scenarios with varying camera parameters.

Three-dimensional visible light positioning through fisheye camera

Abstract Visible light can be utilized to achieve indoor high-precision positioning. However, most visible light positioning (VLP) systems only achieve two-dimensional (2D) positioning and require a dense LED layout as the transmitter. Therefore, we propose a novel three-dimensional (3D) visible light indoor positioning system, which uses the fisheye camera as the receiver, designs an LED width comparison method to reduce the requirement of LED density, and realizes 3D positioning. When images captured by the fisheye camera distort, we use the checkerboard calibration method to correct the distortion parameters of the fisheye camera. The parameters can be used to improve the impact of image distortion on the positioning effect. For the LED width comparison method, we set up a discrete height-width database, fit a curve between height and LED width, and calculate the receiver height according to the linear relationship. Extensive results show that the positioning system can achieve a 3D positioning with 9.66cm average error.

Automated body measurement of beef cattle based on keypoint detection and local point cloud clustering

Abstract Body size parameters of beef cattle are crucial for assessing growth status and breeding value. In actual farming environments, the various postures of beef cattle and complex backgrounds can affect the accuracy and stability of non-contact body measurement methods. Therefore, this paper proposes a novel method called the cattle body measurement method (CBMM), which combines keypoint detection with local point cloud clustering. First, a keypoint detection model based on YOLOv8-SimBiFPN is constructed. This model enhances the feature extraction and fusion capabilities of YOLOv8-pose by introducing SimAM and BiFPN into the backbone and neck networks, respectively, and realizes 2D keypoint detection for beef cattle in various postures. Second, a 3D keypoint-locating algorithm based on Density-based spatial clustering of applications with noise (DBSCAN) is proposed. This algorithm utilizes 2D keypoints, depth maps and camera parameters to generate local point clouds, which are then clustered using DBSCAN to segment cattle body point clouds, thereby relocating the 3D keypoints based on their positional features. Finally, body size parameters are calculated based on the 3D keypoints and distance formulae. In our experiment, the mean average precision (mAP@0.5) of YOLOv8-SimBiFPN reached 99.1% on an Angus beef cattle keypoint detection dataset. The mean absolute percentage errors for measuring beef cattle withers height, hip height, body depth, body length, and oblique body length using the CBMM were 4.37%, 4.96%, 6.47%, 4.84%, and 4.14%, respectively. In summary, our method can achieve non-contact body measurement for beef cattle in a free-moving state with high accuracy and stability.

Self-Supervised 3D Semantic Occupancy Prediction from Multi-View 2D Surround Images

An accurate 3D representation of the geometry and semantics of an environment builds the basis for a large variety of downstream tasks and is essential for autonomous driving related tasks such as path planning and obstacle avoidance. The focus of this work is put on 3D semantic occupancy prediction, i.e., the reconstruction of a scene as a voxel grid where each voxel is assigned both an occupancy and a semantic label. We present a Convolutional Neural Network-based method that utilizes multiple color images from a surround-view setup with minimal overlap, together with the associated interior and exterior camera parameters as input, to reconstruct the observed environment as a 3D semantic occupancy map. To account for the ill-posed nature of reconstructing a 3D representation from monocular 2D images, the image information is integrated over time: Under the assumption that the camera setup is moving, images from consecutive time steps are used to form a multi-view stereo setup. In exhaustive experiments, we investigate the challenges presented by dynamic objects and the possibilities of training the proposed method with either 3D or 2D reference data. Latter being motivated by the comparably higher costs of generating and annotating 3D ground truth data. Moreover, we present and investigate a novel self-supervised training scheme that does not require any geometric reference data, but only relies on sparse semantic ground truth. An evaluation on the Occ3D dataset, including a comparison against current state-of-the-art self-supervised methods from the literature, demonstrates the potential of our self-supervised variant.

A calibration method for telecentric line-scan cameras with an enhanced dynamic imaging model and initial estimates derived from direct linear transformation

This study presents a calibration method for telecentric line-scan cameras, incorporating a dynamic imaging model and initial estimates derived from direct linear transformation. The enhanced dynamic imaging model includes an inclination parameter instead of velocities in three orthogonal spatial directions. It is applicable to any direction of motion, considers lens distortion, and accommodates the line sensor at an arbitrary position relative to the optical axis. Employing the enhanced camera model, the initial values of telecentric line-scan camera parameters are determined derived from direct linear transformation. Sensitivity of our method with respect to the noise level of image points and the number of pattern planes are assessed through numerical simulations, while effectiveness and robustness are validated through experiments using real-world data, achieving a reprojection error of 0.6386 pixels and a standard deviation of 0.0160 pixels. The versatility of the proposed method can be extended to measurement applications utilizing flatbed scanners.

Active dosimetry for VHEE FLASH radiotherapy using beam profile monitors and charge measurements

The discovery of the FLASH effect has revealed a high potential for treating cancer more efficiently by sparing healthy tissue. The surge in related medical research activities over the last couple of years has triggered a demand for technology with the capability of generating and measuring ionizing radiation at ultra-high dose-rates (UHDR). A reliable dosimetry system is an integral part of a radiotherapy machine. Because existing active dosimetry methods are unable to handle the dose-rates required for FLASH, UHDR dosimetry has emerged as an important area of research. In this paper we present an active dosimetry method based on a scintillating screen and an integrating current transformer. This method provides a simultaneous measurement of the absolute dose delivery as well as the 2D dose distribution. The measurements have been correlated with corresponding readings from radiochromic films (RCFs), and a procedure for image processing has been established. Moreover, different methods of calibrating the active dosimetry system against RCFs have been introduced and evaluated. Lastly, we present results which demonstrate that an agreement with RCFs of better than 5% can be realistically expected if camera parameters are carefully optimized.

Analytical equation for camera imaging with refractive interfaces

Camera imaging through refractive interfaces is a crucial issue in photogrammetric measurements. Most past studies adopted numerical optimization algorithms based on refractive ray tracing procedures. In these studies, the camera and interface parameters are usually calculated iteratively with numerical optimization algorithms. Inappropriate initial values can cause iterations to diverge. Meanwhile, these iterations cannot efficiently reveal the accurate nature of refractive imaging. Therefore, obtaining camera calibration results that are both flexible and physically interpretable continues to be challenging. Consequently, in this study, we modeled refractive imaging by employing ray transfer matrix analysis. Subsequently, we deduced an analytical refractive imaging (ARI) equation that explicitly describes the refractive geometry in a matrix form. Although this equation is built upon the paraxial approximation, we executed a numerical experiment that shows that the developed analytical equation can accurately illustrate refractive imaging with a considerable object distance and a slightly tilted angle of the flat interface. This ARI equation can be used to define the expansion center and the normal vector of the flat interface. Finally, we also propose a flexible measurement method to determine the orientation of the flat interface, wherein the orientation can be measured rather than calculated by iterative procedures.

Development of multi-view 3D reconstruction system for bubble flow measurement

This work introduces the development of bubble measurement method utilizing a three-dimensional (3D) reconstruction technique from multi-view images. Multiple synchronized cameras were positioned around a water container, and calibration was performed to obtain external and internal camera parameters. Images of bubbles emerging from a nozzle were captured, and a machine learning technique was used to extract bubble silhouettes as foreground probability distributions, enabling the extraction of bubbles from images obtained with a simple lighting setup. These distributions were then projected onto a 3D voxel space using the visual hull method. It was confirmed that the method can successfully capture bubble generation, detachment, and rise behaviors, offering insights into understanding bubble dynamics and potential applications in 3D computational fluid dynamics validation.

Multi-camera Simultaneous Total Internal Reflection and Interference Reflection Microscopy.

Interference Reflection Microscopy (IRM) is an optical technique that relies on the interference between the reflected light from an incident beam as it passes through materials of different refractive indices. This technique has been successfully used to image microtubules, biologically important biofilaments with a diameter of 25 nm. However, it is often desirable to image both the microtubule and microtubule interacting proteins simultaneously. Here we present a simple modification to a standard multi-color total internal reflection fluorescence (TIRF) microscope that enables simultaneous high-speed IRM and single molecule TIRF imaging. Our design utilizes a camera for each channel (IRM and TIRF) allowing independent optimization of camera parameters for the two different modalities. We illustrate its application by imaging unlabeled microtubules and GFP-labeled end-binding protein EB1 which forms comets on the tips of polymerizing microtubules. Our design is easily implemented, and with minimal cost, making it accessible to any laboratory with an existing fluorescence microscope.

3D Positioning of Drones through Images.

Drones traditionally rely on satellite signals for positioning and altitude. However, when in a special denial environment, satellite communication is interrupted, and the traditional positioning and height determination methods face challenges. We made a dataset at the height of 80-200 m and proposed a multi-scale input network. The positioning index RDS achieved 76.3 points, and the positioning accuracy within 20 m was 81.7%. This paper proposes a method to judge the height by image alone, without the support of other sensor data. One height judgment can be made per single image. Based on the UAV image-satellite image matching positioning technology, by calculating the actual area represented by the UAV image in real space, combined with the fixed parameters of the optical camera, the actual height of the UAV flight is calculated, which is 80-200 m, and the relative error rate of height is 18.1%.

Deep learning-based perspective distortion correction for outdoor photovoltaic module images

To address the growing demand for photovoltaic inspection, an increasing volume of photoluminescence, electroluminescence, and infrared image datasets is being produced for research and analysis. Most of these images exhibit perspective distortions, which introduce challenges in subsequent analysis, such as defect detection and classification. Hence, distortion correction is required during the preprocessing stage. However, manually correcting each dataset to eliminate distortions is labour-intensive. This paper addresses this issue by developing a convolutional neural network model to estimate the camera parameters of yaw, pitch, and roll. A heuristic method was referenced as a comparative benchmark, demonstrating that the developed deep learning-based approach is equally accurate and significantly faster. The findings underscore the potential of deep learning techniques in automating and refining image analysis in photovoltaic diagnostics, offering substantial improvements over traditional methods in terms of speed and scalability.

Spectrofluorimetric and smartphone-based detection methods for determination of gentamicin

This paper presents the development of spectrofluorimetric and smartphone-based detection methods for gentamicin determination using fluorescamine as a reagent. The research included selecting excitation (415 nm) and emission (489 nm) wavelengths, reaction time, and conditions like reagent concentration, and pH of the reaction medium. Moreover, the optimal operating parameters of the smartphone camera, like ISO, white balance, camera shutter, and RGB model channel were selected. Analytical parameters of the developed spectrofluorimetric and smartphone-based methods were estimated including the linear range: 0.04–15.00 mg dm−3 and 0.18–1.20 mg dm−3, respectively, limits of detection and quantification: 0.01 and 0.04 mg dm−3, and 0.06 and 0.18 mg dm−3, respectively, and precision (CV, n = 6): 5.2% and 2.8%, respectively. The proposed approaches were successfully applied to determine gentamicin in pharmaceutical samples. The obtained results were consistent with values declared by manufacturers and satisfactory recovery values, 93.2–113.6% were obtained for both spectrofluorimetric and smartphone-based methods. The developed fluorimetric method with smartphone-based detection provides a low limit of detection specific to spectrofluorimetric methods whereas the measurement system is a simple, easily accessible, compact, and low-cost device. Hence, it can become a competitive alternative to other gentamicin determination methods.Graphical abstract

3DGEN: a framework for generating custom-made synthetic 3D datasets for civil structure health monitoring

The availability of high-quality datasets is increasingly critical in the field of computer vision-based civil structural health monitoring, where deep learning approaches have gained prominence. However, the lack of specialized datasets for such tasks poses a significant challenge for training a reliable model. To address this challenge, a framework, 3DGEN, is proposed to swiftly generate realistic synthetic 3D datasets which can be targeted for specific tasks. The framework is based on diverse 3D civil structural models, rendering them from various angles and providing depth information and camera parameters for training neural networks. By employing mathematical methods, such as analytical solutions and/or numerical simulations, deformation of civil engineering structures can be generated, ensuring a reliable representation of their real-world shapes and characteristics in the 3D datasets. For texture generation, a generative 3D texturing method enables users to specify desired textures using plain English sentences. Two successful experiments are conducted to (1) assess the efficiency of generating the 3D datasets using two distinct structures, (2) train a monocular depth estimation network to perform 3D surface reconstruction with the generated dataset. Notably, 3DGEN is not limited to 3D surface reconstruction; it can also be used for training neural networks for various other tasks. The code and dataset are available at: https://github.com/YANDA-SHAO/Beam-Dataset-SE

Effects of camera calibration on the accuracy of Unmanned Aerial Vehicle sensor products

Utilization of Unmanned Aerial Vehicles (UAV) mounted with non-metric consumer grade digital cameras is on the rise globally due to their affordability and ease of operation. For high-accuracy UAV products, there is a need for accurate camera parameters determined through camera calibration. Camera calibration can be performed before (pre-calibration) or during the bundle block adjustment (self-calibration). This study aims to analyze the effect of camera calibration parameters on the accuracy of UAV products namely Digital Elevation Model (DEM) and orthoimage. Camera calibration parameters are estimated using two approaches namely self-calibration which deploys 3D image information of the scene in a bundle adjustment and a 2D reference object-based approach known as Zhang’s technique which requires image information of a planar pattern. A DJI FC220 camera mounted on a DJI Mavic Pro UAV was used. Self-calibration was deployed in Agisoft Metashape software based on Brown’s method and Zhang’s technique was deployed in MATLAB and OpenCV. Based on internal measures of accuracy, OpenCV yields the least reprojection error of 0.14 followed by MATLAB (0.79) and self-calibration (1.21). Processing without calibration yields the highest reprojection error of 2.18. Based on external measures of accuracy, that is the geometric accuracy of UAV products, self-calibration yields the least RMSE of 8.2 and 1.4 cm, for the horizontal and vertical, respectively, followed by Zhang’s technique with 9.6 and 2.3 cm in MATLAB and 13.5 and 4.3 cm in OpenCV. Processing without calibration yields the highest vertical RMSE of 20.0 and 22.9 cm for the horizontal and vertical, respectively. Comparison of accuracy of UAV mapping products computed with and without calibration emphasizes the need for camera calibration to optimize accuracy of UAV products. This study recommends assessing other photogrammetric mapping software and camera calibration approaches and the effect of flying heights on calibration parameters and mapping accuracy.

Motion-error-free calibration of event camera systems using a flashing target

Event cameras, inspired by biological vision, offer high dynamic range, excellent temporal resolution, and minimal data redundancy. Precise calibration of event camera systems is essential for applications such as 3D vision. The cessation of extra gray frame production in popular models like the dynamic vision sensor (DVS) poses significant challenges to achieving high-accuracy calibration. Traditional calibration methods, which rely on motion to trigger events, are prone to movement-related errors. This paper introduces a motion-error-free calibration method for event cameras using a flashing target produced by a standard electronic display that elicits high-fidelity events. We propose an improved events-accumulator to reconstruct gray images with distinct calibration features and develop an optimization method that adjusts camera parameters and control point positions simultaneously, enhancing the calibration accuracy of event camera systems. Experimental results demonstrated higher accuracy compared to the traditional motion-based calibration method (reprojection error: 0.03 vs. 0.96 pixels). The 3D reconstruction error remained around 0.15 mm, significantly improving over the motion-based method’s 8.00 mm. Additionally, the method’s adaptability for hybrid calibration in event-based stereovision systems was verified (e.g., with frame cameras or projectors).

On the Parameter Calibration Method for Dual-Spectral Triocular Camera

The dual-spectrum triocular camera system composed of a binocular visible light camera and mid-infrared camera is used for high-precision thermal fault detection and thermal field reconstruction of industrial equipment. The realization of its function depends on the high-precision camera parameter calibration. The difficulty lies in how to realize the infrared camera calibration quickly and improve the parameter accuracy of multi-lens camera. In this paper, according to the characteristics of the dual-spectral triocular camera system, the theoretical model is constructed, and the circular asymmetric calibration plate under infrared supplementary light is selected as the calibration object through experiments. A method for combining infrared image adaptive histogram enhancement and binarization processing based on the Sauvola algorithm is proposed to effectively calibrate the infrared camera. A global parameter optimization method based on traditional passive vision binocular stereo calibration is proposed. The optimization of experimental parameters finds the reprojection error value is 0.16, which meets the demand of high-precision calibration and solves the problem of difficult-to-calibrate weak image texture of infrared camera and the calibration accuracy of the camera under different imaging capabilities.