Camera Parameters Research Articles

Multicamera simultaneous total internal reflection and interference reflection microscopy.

Interference Reflection Microscopy (IRM) is an optical technique that relies on the interference between the reflected light from an incident beam as it passes through materials of different refractive indices. This technique has been successfully used to image microtubules, biologically important biofilaments with a diameter of 25nm. However, it is often desirable to image both the microtubule and microtubule interacting proteins simultaneously. Here we present a simple modification to a standard multicolour total internal reflection fluorescence (TIRF) microscope that enables simultaneous high-speed IRM and single molecule TIRF imaging. Our design utilises a camera for each channel (IRM and TIRF) allowing independent optimisation of camera parameters for the two different modalities. We illustrate its application by imaging unlabelled microtubules and GFP-labelled end-binding protein EB1, which forms comets on the tips of polymerising microtubules. Our design is easily implemented, and with minimal cost, making it accessible to any laboratory with an existing fluorescence microscope.

NeRFtrinsic Four: An end-to-end trainable NeRF jointly optimizing diverse intrinsic and extrinsic camera parameters

Novel view synthesis using neural radiance fields (NeRF) is the state-of-the-art technique for generating high-quality images from novel viewpoints. Existing methods require a priori knowledge about extrinsic and intrinsic camera parameters. This limits their applicability to synthetic scenes, or real-world scenarios with the necessity of a preprocessing step. Current research on the joint optimization of camera parameters and NeRF focuses on refining noisy extrinsic camera parameters and often relies on the preprocessing of intrinsic camera parameters. Further approaches are limited to cover only one single camera intrinsic. To address these limitations, we propose a novel end-to-end trainable approach called NeRFtrinsic Four. We utilize Gaussian Fourier features to estimate extrinsic camera parameters and dynamically predict varying intrinsic camera parameters through the supervision of the projection error. Our approach outperforms existing joint optimization methods on LLFF and BLEFF. In addition to these existing datasets, we introduce a new dataset called iFF with varying intrinsic camera parameters. NeRFtrinsic Four is a step forward in joint optimization NeRF-based view synthesis and enables more realistic and flexible rendering in real-world scenarios with varying camera parameters.

Efficient class‐agnostic obstacle detection for UAV‐assisted waterway inspection systems

AbstractEnsuring the safety of water airport runways is essential for the correct operation of seaplane flights. Among other tasks, airport operators must identify and remove various objects that may have drifted into the runway area. In this paper, the authors propose a complete and embedded‐friendly waterway obstacle detection pipeline that runs on a camera‐equipped drone. This system uses a class‐agnostic version of the YOLOv7 detector, which is capable of detecting objects regardless of its class. Additionally, through the usage of the GPS data of the drone and camera parameters, the location of the objects are pinpointed with 0.58 m Distance Root Mean Square. In our own annotated dataset, the system is capable of generating alerts for detected objects with a recall of 0.833 and a precision of 1.

360-degree Human Video Generation with 4D Diffusion Transformer

We present a novel approach for generating 360-degree high-quality, spatiotemporally coherent human videos from a single image. Our framework combines the strengths of diffusion transformers for capturing global correlations across viewpoints and time, and CNNs for accurate condition injection. The core is a hierarchical 4D transformer architecture that factorizes self-attention across views, time steps, and spatial dimensions, enabling efficient modeling of the 4D space. Precise conditioning is achieved by injecting human identity, camera parameters, and temporal signals into the respective transformers. To train this model, we collect a multi-dimensional dataset spanning images, videos, multi-view data, and limited 4D footage, along with a tailored multi-dimensional training strategy. Our approach overcomes the limitations of previous methods based on generative adversarial networks or vanilla diffusion models, which struggle with complex motions, viewpoint changes, and generalization. Through extensive experiments, we demonstrate our method's ability to synthesize 360-degree realistic, coherent human motion videos, paving the way for advanced multimedia applications in areas such as virtual reality and animation.

Three-dimensional reconstruction of asphalt road surface texture based on binocular stereo vision

The texture of asphalt pavement plays a pivotal role in ensuring the safe operation of roads. This paper presents a binocular stereo vision hardware system that is designed to acquire asphalt pavement texture features in an economical and efficient manner. The system comprises an image acquisition device, a micro meter, and a camera bracket. The binocular camera baseline length is first determined according to the adjusted camera parameters, with the objective of enhancing the quality of the acquired images. Secondly, the camera calibration was carried out using Matlab and Python software, and the camera parameters were obtained to perform aberration and stereo corrections on the binocular image pairs. Finally, based on Python software, the sum of absolute differences, symmetric diagonal dominance, zero-mean normalised cross-correlation and semi-global block matching algorithms were applied to stereo-match the texture information image set and reconstructed the 3D asphalt pavement point cloud based on parallax mapping. However, it should be noted that the experiment does not consider the effect of light intensity in indoor and outdoor environments. This paper provides a preliminary summary of the current structural design and experiments, and further discusses ideas for subsequent research and improvement. Research and experiments show that the binocular stereo vision imaging system proposed in this paper can effectively match binocular images and present 3D information of asphalt pavement texture. When the binocular camera baseline distance is 40 mm, the maximum calibration error and the average calibration error are both the smallest, which enhances the stability of binocular matching. Furthermore, the camera parameters calculated by the Matlab toolbox calibration are more stable than those obtained by OpenCV calibration. In the indoor experimental environment, the running speeds of the four stereo matching algorithms were compared. In the indoor test environment, the SGBM algorithm was the fastest, with a running speed of 3 seconds, in the order of SGBM>SAD>SSD>ZNCC. The SGBM algorithm demonstrated the most effective parallax mapping reconstruction of asphalt pavement point clouds. The reconstruction of pavement texture may be employed in the future for the real-time monitoring of asphalt pavement skid resistance.

Research on a Method for Measuring the Pile Height of Materials in Agricultural Product Transport Vehicles Based on Binocular Vision.

The advancement of unloading technology in combine harvesting is crucial for the intelligent development of agricultural machinery. Accurately measuring material pile height in transport vehicles is essential, as uneven accumulation can lead to spillage and voids, reducing loading efficiency. Relying solely on manual observation for measuring stack height can decrease harvesting efficiency and pose safety risks due to driver distraction. This research applies binocular vision to agricultural harvesting, proposing a novel method that uses a stereo matching algorithm to measure material pile height during harvesting. By comparing distance measurements taken in both empty and loaded states, the method determines stack height. A linear regression model processes the stack height data, enhancing measurement accuracy. A binocular vision system was established, applying Zhang's calibration method on the MATLAB (R2019a) platform to correct camera parameters, achieving a calibration error of 0.15 pixels. The study implemented block matching (BM) and semi-global block matching (SGBM) algorithms using the OpenCV (4.8.1) library on the PyCharm (2020.3.5) platform for stereo matching, generating disparity, and pseudo-color maps. Three-dimensional coordinates of key points on the piled material were calculated to measure distances from the vehicle container bottom and material surface to the binocular camera, allowing for the calculation of material pile height. Furthermore, a linear regression model was applied to correct the data, enhancing the accuracy of the measured pile height. The results indicate that by employing binocular stereo vision and stereo matching algorithms, followed by linear regression, this method can accurately calculate material pile height. The average relative error for the BM algorithm was 3.70%, and for the SGBM algorithm, it was 3.35%, both within the acceptable precision range. While the SGBM algorithm was, on average, 46 ms slower than the BM algorithm, both maintained errors under 7% and computation times under 100 ms, meeting the real-time measurement requirements for combine harvesting. In practical operations, this method can effectively measure material pile height in transport vehicles. The choice of matching algorithm should consider container size, material properties, and the balance between measurement time, accuracy, and disparity map completeness. This approach aids in manual adjustment of machinery posture and provides data support for future autonomous master-slave collaborative operations in combine harvesting.

A Physics-Informed Neural Network Approach to Augmented Dynamics Visual Servoing of Multirotors.

This article presents a visual servoing strategy that integrates the capabilities of a physics-informed neural network (PINN) to estimate system uncertainties and inaccuracies with a dynamics-centered visual servoing technique for multirotors. The proposed method effectively combines these approaches, eliminating the need for inverse Jacobian calculations to determine multirotor motion by directly relating pixel variations to the multirotor's torque and thrust inputs, while also strengthening the method's robustness through the utilization of the PINN to model and address uncertainties in camera and multirotor parameters, as well as the modeling inaccuracies inherent in the dynamics-centered visual servoing technique. In contrast to existing state-of-the-art data-driven approaches, the proposed PINN approach requires, on average, 65% less labeled data to characterize uncertainties and inaccuracies. To ensure real-time implementation of the visual servoing model, the PINN-learned model is combined with an adaptive horizon monotonically weighted nonlinear model predictive controller (NMPC), capable of processing control efforts at rates 10 times faster than existing Tube MPC and Adaptive MPC strategies. These findings are validated through real-time trajectory tracking experiments, which not only highlight the effectiveness of the proposed approach in approximating modeling inaccuracies but also its capability in handling uncertainties upto 70% in camera parameters.

Seamless and Aligned Texture Optimization for 3D Reconstruction

AbstractRestoring the appearance of the model is a crucial step for achieving realistic 3D reconstruction. High‐fidelity textures can also conceal some geometric defects. Since the estimated camera parameters and reconstructed geometry usually contain errors, subsequent texture mapping often suffers from undesirable visual artifacts such as blurring, ghosting, and visual seams. In particular, significant misalignment between the reconstructed model and the registered images will lead to texturing the mesh with inconsistent image regions. However, eliminating various artifacts to generate high‐quality textures remains a challenge. In this paper, we address this issue by designing a texture optimization method to generate seamless and aligned textures for 3D reconstruction. The main idea is to detect misalignment regions between images and geometry and exclude them from texture mapping. To handle the texture holes caused by these excluded regions, a cross‐patch texture hole‐filling method is proposed, which can also synthesize plausible textures for invisible faces. Moreover, for better stitching of the textures from different views, an improved camera pose optimization is present by introducing color adjustment and boundary point sampling. Experimental results show that the proposed method can eliminate the artifacts caused by inaccurate input data robustly and produce high‐quality texture results compared with state‐of‐the‐art methods.

Online extrinsic parameters calibration of on-board stereo cameras based on certifiable optimization

The extrinsic parameters of on-board stereo cameras can be slightly altered due to temperature fluctuations, vibrations, and accidental impacts during automobile driving, leading to significant performance loss in dense stereo matching. In this paper, we propose an online calibration method based on certifiable optimization to address this issue. Initially, sparse feature points are collected based on plane distribution and disparity. A robust optimization model is then developed to minimize the epipolar error, utilizing iterative local optimization to eliminate outliers and determine the 5DOF extrinsic parameters. Subsequently, a relaxation problem is constructed using inliers, and global optimization is performed to certify that the locally optimal results are indeed globally optimal. Comparative experimental results demonstrate that the proposed method offers high accuracy and reliability. Additionally, the quality of the disparity map generated by our calibration method is comparable to that achieved through offline calibration.

Real-time robust liver and gallbladder segmentation during laparoscopic cholecystectomy using convolutional neural networks: an analysis

Aim: Images in different laparoscopic cholecystectomy datasets are acquired using various camera models, parameters, and settings, with the annotation methods varying by institution. These factors result in inconsistent inference performance of the network model. This study aims to identify the optimal network model architecture for liver and gallbladder segmentation from several options. Then, the performance and robustness of the optimal network model are evaluated using an independent dataset that is not included in the training. Methods: The public dataset, CholecSeg8k, was utilized as the input for the network model training, validation, and testing. A local private dataset from KPJ Damansara Hospital, Selangor, Malaysia, was used for testing purposes only. For the implementation of liver and gallbladder segmentation, segmentation models, a public Python library was employed. Results: Among the experiments, highly accurate liver and gallbladder segmentation results were achieved using the feature pyramid network (FPN) architecture as the network model, with the Inception-ResNet-v2 architecture as the network backbone. The best-trained network model resulted in a loss of 0.070955, a mean intersection over union (IoU) score of 0.95896, and a mean F1-score of 0.9773 on the test set. However, visualized results for the private dataset contained considerable false-negative areas. Conclusion: The proposed automated technique has the potential to serve as an alternative to the conventional indocyanine green injection along with near-infrared fluorescence imaging (ICG-NIRF)-based method for liver and gallbladder segmentation during laparoscopic cholecystectomy. Future work will focus on enhancing the results of the private dataset. Additionally, a surgeon-assistant robotic arm that will use the liver and gallbladder segmentation results for camera steering will be analyzed.

Pupil-centric imaging model for zoom camera calibration.

Zoom camera calibration has always been challenging, as arbitrary zoom/focus settings change the camera parameters. Current calibration methods are based on the pinhole imaging model, which results in coupled calibration parameters due to the lack of geometric/physical constraints. In this Letter, we present a novel, to the best of our knowledge, pupil-centric imaging model that accounts for the camera's radial, decentering, and mustache distortions at various zoom settings using exit pupil offsets and a non-frontal lens-sensor model. Therefore, we provide a reasonable physical explanation for the different distortion effects. Global optimization is performed based on the proposed initial camera calibration and bundle adjustment under several zoom and autofocus setting combinations. Experiments using three representative zoom cameras demonstrate the effectiveness of the proposed method. Its relative measurement accuracy is better than that of current state-of-the-art methods.

Three-dimensional visible light positioning through fisheye camera

Abstract Visible light can be utilized to achieve indoor high-precision positioning. However, most visible light positioning (VLP) systems only achieve two-dimensional (2D) positioning and require a dense LED layout as the transmitter. Therefore, we propose a novel three-dimensional (3D) visible light indoor positioning system, which uses the fisheye camera as the receiver, designs an LED width comparison method to reduce the requirement of LED density, and realizes 3D positioning. When images captured by the fisheye camera distort, we use the checkerboard calibration method to correct the distortion parameters of the fisheye camera. The parameters can be used to improve the impact of image distortion on the positioning effect. For the LED width comparison method, we set up a discrete height-width database, fit a curve between height and LED width, and calculate the receiver height according to the linear relationship. Extensive results show that the positioning system can achieve a 3D positioning with 9.66cm average error.

Automated body measurement of beef cattle based on keypoint detection and local point cloud clustering

Abstract Body size parameters of beef cattle are crucial for assessing growth status and breeding value. In actual farming environments, the various postures of beef cattle and complex backgrounds can affect the accuracy and stability of non-contact body measurement methods. Therefore, this paper proposes a novel method called the cattle body measurement method (CBMM), which combines keypoint detection with local point cloud clustering. First, a keypoint detection model based on YOLOv8-SimBiFPN is constructed. This model enhances the feature extraction and fusion capabilities of YOLOv8-pose by introducing SimAM and BiFPN into the backbone and neck networks, respectively, and realizes 2D keypoint detection for beef cattle in various postures. Second, a 3D keypoint-locating algorithm based on Density-based spatial clustering of applications with noise (DBSCAN) is proposed. This algorithm utilizes 2D keypoints, depth maps and camera parameters to generate local point clouds, which are then clustered using DBSCAN to segment cattle body point clouds, thereby relocating the 3D keypoints based on their positional features. Finally, body size parameters are calculated based on the 3D keypoints and distance formulae. In our experiment, the mean average precision (mAP@0.5) of YOLOv8-SimBiFPN reached 99.1% on an Angus beef cattle keypoint detection dataset. The mean absolute percentage errors for measuring beef cattle withers height, hip height, body depth, body length, and oblique body length using the CBMM were 4.37%, 4.96%, 6.47%, 4.84%, and 4.14%, respectively. In summary, our method can achieve non-contact body measurement for beef cattle in a free-moving state with high accuracy and stability.

Self-Supervised 3D Semantic Occupancy Prediction from Multi-View 2D Surround Images

An accurate 3D representation of the geometry and semantics of an environment builds the basis for a large variety of downstream tasks and is essential for autonomous driving related tasks such as path planning and obstacle avoidance. The focus of this work is put on 3D semantic occupancy prediction, i.e., the reconstruction of a scene as a voxel grid where each voxel is assigned both an occupancy and a semantic label. We present a Convolutional Neural Network-based method that utilizes multiple color images from a surround-view setup with minimal overlap, together with the associated interior and exterior camera parameters as input, to reconstruct the observed environment as a 3D semantic occupancy map. To account for the ill-posed nature of reconstructing a 3D representation from monocular 2D images, the image information is integrated over time: Under the assumption that the camera setup is moving, images from consecutive time steps are used to form a multi-view stereo setup. In exhaustive experiments, we investigate the challenges presented by dynamic objects and the possibilities of training the proposed method with either 3D or 2D reference data. Latter being motivated by the comparably higher costs of generating and annotating 3D ground truth data. Moreover, we present and investigate a novel self-supervised training scheme that does not require any geometric reference data, but only relies on sparse semantic ground truth. An evaluation on the Occ3D dataset, including a comparison against current state-of-the-art self-supervised methods from the literature, demonstrates the potential of our self-supervised variant.

A calibration method for telecentric line-scan cameras with an enhanced dynamic imaging model and initial estimates derived from direct linear transformation

This study presents a calibration method for telecentric line-scan cameras, incorporating a dynamic imaging model and initial estimates derived from direct linear transformation. The enhanced dynamic imaging model includes an inclination parameter instead of velocities in three orthogonal spatial directions. It is applicable to any direction of motion, considers lens distortion, and accommodates the line sensor at an arbitrary position relative to the optical axis. Employing the enhanced camera model, the initial values of telecentric line-scan camera parameters are determined derived from direct linear transformation. Sensitivity of our method with respect to the noise level of image points and the number of pattern planes are assessed through numerical simulations, while effectiveness and robustness are validated through experiments using real-world data, achieving a reprojection error of 0.6386 pixels and a standard deviation of 0.0160 pixels. The versatility of the proposed method can be extended to measurement applications utilizing flatbed scanners.

Active dosimetry for VHEE FLASH radiotherapy using beam profile monitors and charge measurements

The discovery of the FLASH effect has revealed a high potential for treating cancer more efficiently by sparing healthy tissue. The surge in related medical research activities over the last couple of years has triggered a demand for technology with the capability of generating and measuring ionizing radiation at ultra-high dose-rates (UHDR). A reliable dosimetry system is an integral part of a radiotherapy machine. Because existing active dosimetry methods are unable to handle the dose-rates required for FLASH, UHDR dosimetry has emerged as an important area of research. In this paper we present an active dosimetry method based on a scintillating screen and an integrating current transformer. This method provides a simultaneous measurement of the absolute dose delivery as well as the 2D dose distribution. The measurements have been correlated with corresponding readings from radiochromic films (RCFs), and a procedure for image processing has been established. Moreover, different methods of calibrating the active dosimetry system against RCFs have been introduced and evaluated. Lastly, we present results which demonstrate that an agreement with RCFs of better than 5% can be realistically expected if camera parameters are carefully optimized.

Analytical equation for camera imaging with refractive interfaces

Camera imaging through refractive interfaces is a crucial issue in photogrammetric measurements. Most past studies adopted numerical optimization algorithms based on refractive ray tracing procedures. In these studies, the camera and interface parameters are usually calculated iteratively with numerical optimization algorithms. Inappropriate initial values can cause iterations to diverge. Meanwhile, these iterations cannot efficiently reveal the accurate nature of refractive imaging. Therefore, obtaining camera calibration results that are both flexible and physically interpretable continues to be challenging. Consequently, in this study, we modeled refractive imaging by employing ray transfer matrix analysis. Subsequently, we deduced an analytical refractive imaging (ARI) equation that explicitly describes the refractive geometry in a matrix form. Although this equation is built upon the paraxial approximation, we executed a numerical experiment that shows that the developed analytical equation can accurately illustrate refractive imaging with a considerable object distance and a slightly tilted angle of the flat interface. This ARI equation can be used to define the expansion center and the normal vector of the flat interface. Finally, we also propose a flexible measurement method to determine the orientation of the flat interface, wherein the orientation can be measured rather than calculated by iterative procedures.

Development of multi-view 3D reconstruction system for bubble flow measurement

This work introduces the development of bubble measurement method utilizing a three-dimensional (3D) reconstruction technique from multi-view images. Multiple synchronized cameras were positioned around a water container, and calibration was performed to obtain external and internal camera parameters. Images of bubbles emerging from a nozzle were captured, and a machine learning technique was used to extract bubble silhouettes as foreground probability distributions, enabling the extraction of bubbles from images obtained with a simple lighting setup. These distributions were then projected onto a 3D voxel space using the visual hull method. It was confirmed that the method can successfully capture bubble generation, detachment, and rise behaviors, offering insights into understanding bubble dynamics and potential applications in 3D computational fluid dynamics validation.

Multi-camera Simultaneous Total Internal Reflection and Interference Reflection Microscopy.

Interference Reflection Microscopy (IRM) is an optical technique that relies on the interference between the reflected light from an incident beam as it passes through materials of different refractive indices. This technique has been successfully used to image microtubules, biologically important biofilaments with a diameter of 25 nm. However, it is often desirable to image both the microtubule and microtubule interacting proteins simultaneously. Here we present a simple modification to a standard multi-color total internal reflection fluorescence (TIRF) microscope that enables simultaneous high-speed IRM and single molecule TIRF imaging. Our design utilizes a camera for each channel (IRM and TIRF) allowing independent optimization of camera parameters for the two different modalities. We illustrate its application by imaging unlabeled microtubules and GFP-labeled end-binding protein EB1 which forms comets on the tips of polymerizing microtubules. Our design is easily implemented, and with minimal cost, making it accessible to any laboratory with an existing fluorescence microscope.

3D Positioning of Drones through Images.

Drones traditionally rely on satellite signals for positioning and altitude. However, when in a special denial environment, satellite communication is interrupted, and the traditional positioning and height determination methods face challenges. We made a dataset at the height of 80-200 m and proposed a multi-scale input network. The positioning index RDS achieved 76.3 points, and the positioning accuracy within 20 m was 81.7%. This paper proposes a method to judge the height by image alone, without the support of other sensor data. One height judgment can be made per single image. Based on the UAV image-satellite image matching positioning technology, by calculating the actual area represented by the UAV image in real space, combined with the fixed parameters of the optical camera, the actual height of the UAV flight is calculated, which is 80-200 m, and the relative error rate of height is 18.1%.