Camera Parameters Research Articles

Research on seamless image stitching based on fast marching method

AbstractImage stitching is an important way to achieve large‐field high‐resolution imaging. The inconsistencies in brightness and structure and defects in ghosting, blurring and misalignment between images, which are inevitable and difficult to eliminate, make a challenge to image stitching, due to the external lighting environment and changes in camera pose and parameters. Here, a novel method is proposed to search for the optimal seamline based on the fast marching method, which can stitch large parallax images with high quality. A feature weight map is first formed based on the similarity in colour, edge, texture and saliency of the images. Then it is used as the cost value of the seamline to search for the optimal seamline by fast marching method. The results show that this new method is more efficient to reduce defects, such as ghosting, misalignment and chromatic aberration, and realize high quality image stitching compared with traditional stitching tools and methods, which provides a new perspective for image stitching technology.

An improved camera model for oblique-viewing laparoscopes: high reprojection accuracy independent of telescope rotation

Objective. Oblique-viewing laparoscopes are popular in laparoscopic surgeries where the target anatomy is located in narrow areas. Their viewing direction can be shifted by telescope rotation without changing the laparoscope pose. This rotation also changes laparoscope camera parameters that are estimated by camera calibration to be able to reproject an anatomical model onto the laparoscopic view, creating augmented reality (AR). The aim of this study was to develop a camera model that accounts for these changes, achieving high reprojection accuracy for any telescope rotation. Approach. Camera parameters were acquired by calibrations encompassing a wide telescope rotation range. For those parameters showing periodic changes upon rotation, interpolation models were created and used to establish an updatable camera model. With this model, corner points of a tracked checkerboard were reprojected onto the checkerboard laparoscopic images, at random rotation angles. Root-mean-square reprojection errors (RMSEs) were calculated between the reprojected and imaged corner points. Main results. Reprojection RMSEs were low and approximately independent on telescope rotation angle, over a wide rotation range of 320°. The mean reprojection RMSE was 2.8 0.7 pixels for a conventional laparoscope and 3.6 0.7 pixels for a chip-on-the-tip (COTT) laparoscope, corresponding to 0.3 0.1 mm and 0.4 0.1 mm in world coordinates respectively. Worst-case reprojection errors were about 9 pixels (0.8 mm) for both laparoscopes. Significance. The camera model developed in this study improves on existing models for oblique-viewing laparoscopes because it provides high reprojection accuracy independent of the telescope rotation angle and is applicable for conventional and chip-on-a-tip oblique-viewing laparoscopes. The work presented here is an important step towards creating accurate AR in image-guided interventions where oblique-viewing laparoscopes are used while simultaneously providing the surgeon the flexibility to rotate the telescope to any desired rotation angle. Acronyms. CC: camera coordinates; CCToolbox: camera calibration toolbox; COTT: chip-on-the-tip; CS: camera sensor; DD: decentering distortion; FL: focal length; OTS: optical tracking system; PP: principal point; RD: radial distortion; SI: supplementary information; hand-eye translation component

Rover Attitude and Camera Parameter: Rock Measurements on Mars Surface Based on Rover Attitude and Camera Parameter for Tianwen-1 Mission

Rocks, prominent features on the surface of Mars, are a primary focus of Mars exploration missions. The accuracy of recognizing rock information, including size and position, deeply affects the path planning for rovers on Mars and the geological exploration of Mars. In this paper, we present a rock measurement method for the Mars surface based on a Rover Attitude and Camera Parameter (RACP). We analyze the imaging process of the Navigation and Terrain Camera (NaTeCam) on the Zhurong rover, which involves utilizing a semi-spherical model (SSM) to characterize the camera’s attitude, a projection model (PM) to connect the image data with the three-dimensional (3D) environment, and then estimating the distance and size of rocks. We conduct a test on NaTeCam images and find that the method is effective in measuring the distance and size to Martian rocks and identifying rocks at specific locations. Furthermore, an analysis of the impact of uncertain factors is conducted. The proposed RACP method offers a reliable solution for automatically analyzing the rocks on Mars, which provides a possible solution for the route planning in similar tasks.

Impact of Wolfmet Tungsten Alloys as Parallel-Hole Collimator Material on Single-Photon Emission Computed Tomography Image Quality and Functional Parameters: A Simulating Medical Imaging Nuclear Detectors Monte Carlo Study.

Objectives Collimators have a significant role in image quality and detectability in single-photon emission computed tomography (SPECT) imaging. Using an appropriate alloy that effectively absorbs scattered photons, without induced secondary x-rays, and with proper rigidity and weight may provide an effective approach to the image improvement that conventionally collimators made of lead (Pb). Materials and Methods A Siemens E.CAM SPECT imaging system equipped with low-energy high-resolution (LEHR) collimator was simulated by the Simulating Medical Imaging Nuclear Detectors Monte Carlo program. Experimental and simulated data were compared based on a 2-mm 99m Tc point source in an acrylic cylindrical Deluxe phantom (Data Spectrum, Inc). Seven types of tungsten (W) alloys (Wolfmet), with W content from 90 to 97% by weight, were then used as collimator materials of the simulated system. Camera parameters, such as energy- and spatial resolution, image contrast, and collimator-related parameters, such as fraction of septal penetration, scatter-to-primary ratios, and percentage of induced secondary x-rays, due to interactions in the collimator, were evaluated. Results Acceptable conformity was found for the simulated and experiment systems in terms of energy spectra, 10.113 and 10.140%, full width at half-maximum (FWHM) of the point spread function (PSF) curves, 8.78 and 9.06 mm, sensitivity, 78.46 and 78.34 cps/MBq, and contrast in images of 19.1 mm cold spheres in the Deluxe phantom, 79.17 and 78.97%, respectively. Results on the parameters of the simulated system with LEHR collimator made from the alloys showed that the alloy consisting of 90% W, 6% nickel, and 4% copper provided an FWHM of 8.76 mm, resulting in a 0.2% improvement in spatial resolution. Furthermore, all the Wolfmet collimators showed a 48% reduction in the amount of X-rays production compared to the Pb. Conclusion A Wolfmet LEHR collimator, made by a combination of W (90%), Ni (6%), and Cu (6%) provides a better image quality and detectability compared to the Pb.

Research on 3D Reconstruction of Binocular Vision Based on Thermal Infrared.

Thermal infrared imaging is less affected by lighting conditions and smoke compared to visible light imaging. However, thermal infrared images often have lower resolution and lack rich texture details, making them unsuitable for stereo matching and 3D reconstruction. To enhance the quality of infrared stereo imaging, we propose an advanced stereo matching algorithm. Firstly, the images undergo preprocessing using a non-local mean noise reduction algorithm to remove thermal noise and achieve a smoother result. Subsequently, we perform camera calibration using a custom-made chessboard calibration board and Zhang's camera calibration method to obtain accurate camera parameters. Finally, the disparity map is generated using the SGBM (semi-global block matching) algorithm based on the weighted least squares method, enabling the 3D point cloud reconstruction of the object. The experimental results demonstrate that the proposed algorithm performs well in objects with sufficient thermal contrast and relatively simple scenes. The proposed algorithm reduces the average error value by 10.9 mm and the absolute value of the average error by 1.07% when compared with the traditional SGBM algorithm, resulting in improved stereo matching accuracy for thermal infrared imaging. While ensuring accuracy, our proposed algorithm achieves the stereo reconstruction of the object with a good visual effect, thereby holding high practical value.

Acousto-optic modulator-based improvement in imaging through scattering media.

Reduced visibility is a common problem when light traverses through a scattering medium, and it becomes difficult to identify an object in such scenarios. What we believe to be a novel proof-of-principle technique for improving image visibility based on the quadrature lock-in discrimination algorithm in which the demodulation is performed using an acousto-optic modulator is presented here. A significant improvement in image visibility is achieved using a series of frames. We have also performed systematic imaging by varying the camera parameters, such as exposure time, frame rate, and series length, to investigate their effect on enhancing image visibility.

Conv-TabNet: an efficient adaptive color correction network for smartphone-based urine component analysis.

The camera function of a smartphone can be used to quantitatively detect urine parameters anytime, anywhere. However, the color captured by different cameras in different environments is different. A method for color correction is proposed for a urine test strip image collected using a smartphone. In this method, the color correction model is based on the color information of the urine test strip, as well as the ambient light and camera parameters. Conv-TabNet, which can focus on each feature parameter, was designed to correct the color of the color blocks of the urine test strip. The color correction experiment was carried out in eight light sources on four mobile phones. The experimental results show that the mean absolute error of the new method is as low as 2.8±1.8, and the CIEDE2000 color difference is 1.5±1.5. The corrected color is almost consistent with the standard color by visual evaluation. This method can provide a technology for the quantitative detection of urine test strips anytime and anywhere.

Adaptable 2D to 3D Stereo Vision Image Conversion Based on a Deep Convolutional Neural Network and Fast Inpaint Algorithm.

Algorithms for converting 2D to 3D are gaining importance following the hiatus brought about by the discontinuation of 3D TV production; this is due to the high availability and popularity of virtual reality systems that use stereo vision. In this paper, several depth image-based rendering (DIBR) approaches using state-of-the-art single-frame depth generation neural networks and inpaint algorithms are proposed and validated, including a novel very fast inpaint (FAST). FAST significantly exceeds the speed of currently used inpaint algorithms by reducing computational complexity, without degrading the quality of the resulting image. The role of the inpaint algorithm is to fill in missing pixels in the stereo pair estimated by DIBR. Missing estimated pixels appear at the boundaries of areas that differ significantly in their estimated distance from the observer. In addition, we propose parameterizing DIBR using a singular, easy-to-interpret adaptable parameter that can be adjusted online according to the preferences of the user who views the visualization. This single parameter governs both the camera parameters and the maximum binocular disparity. The proposed solutions are also compared with a fully automatic 2D to 3D mapping solution. The algorithm proposed in this work, which features intuitive disparity steering, the foundational deep neural network MiDaS, and the FAST inpaint algorithm, received considerable acclaim from evaluators. The mean absolute error of the proposed solution does not contain statistically significant differences from state-of-the-art approaches like Deep3D and other DIBR-based approaches using different inpaint functions. Since both the source codes and the generated videos are available for download, all experiments can be reproduced, and one can apply our algorithm to any selected video or single image to convert it.

ARAI-MVSNet: A multi-view stereo depth estimation network with adaptive depth range and depth interval

Multi-View Stereo (MVS) is a fundamental problem in geometric computer vision which aims to reconstruct a scene using multi-view images with known camera parameters. However, the mainstream approaches represent the scene with a fixed all-pixel depth range and equal depth interval partition, which will result in inadequate utilization of depth planes and imprecise depth estimation. In this paper, we present a novel multi-stage coarse-to-fine framework to achieve adaptive all-pixel depth range and depth interval. We predict a coarse depth map in the first stage, then an Adaptive Depth Range Prediction module is proposed in the second stage to zoom in the scene by leveraging the reference image and the obtained depth map in the first stage and predict a more accurate all-pixel depth range for the following stages. In the third and fourth stages, we propose an Adaptive Depth Interval Adjustment module to achieve adaptive variable interval partition for pixel-wise depth range. The depth interval distribution in this module is normalized by Z-score, which can allocate dense depth hypothesis planes around the potential ground truth depth value and vice versa to achieve more accurate depth estimation. Extensive experiments on four widely used benchmark datasets (DTU, TnT, BlendedMVS, ETH 3D) demonstrate that our model achieves state-of-the-art performance and yields competitive generalization ability. Particularly, our method achieves the highest Acc and Overall on the DTU dataset, while attaining the highest Recall and F1-score on the Tanks and Temples intermediate and advanced dataset. Moreover, our method also achieves the lowest e1 and e3 on the BlendedMVS dataset and the highest Acc and F1-score on the ETH 3D dataset, surpassing all listed methods. Project website: https://github.com/zs670980918/ARAI-MVSNet

Unified shape and appearance reconstruction with joint camera parameter refinement

In this paper, we present an inverse rendering method for the simple reconstruction of shape and appearance of real-world objects from only roughly calibrated RGB images captured under collocated point light illumination. To this end, we gradually reconstruct the lower-frequency geometry information using automatically generated occupancy mask images based on a visual hull initialization of the mesh, to infer the object topology, and a smoothness-preconditioned optimization. By combining this geometry estimation with learning-based SVBRDF parameter inference as well as intrinsic and extrinsic camera parameter refinement in a joint and unified formulation, our novel method is able to reconstruct shape and an isotropic SVBRDF from fewer input images than previous methods. Unlike in other works, we also estimate normal maps as part of the SVBRDF to capture and represent higher-frequency geometric details in a compact way. Furthermore, by regularizing the appearance estimation with a GAN-based SVBRDF generator, we are able to meaningfully limit the solution space. In summary, this leads to a robust automatic reconstruction algorithm for shape and appearance. We evaluated our algorithm on synthetic as well as on real-world data and demonstrate that our method is able to reconstruct complex objects with high-fidelity reflection properties in a robust way, also in the presence of imperfect camera parameter data.

Research on Fruit Spatial Coordinate Positioning by Combining Improved YOLOv8s and Adaptive Multi-Resolution Model

Automated fruit-picking equipment has the potential to significantly enhance the efficiency of picking. Accurate detection and localization of fruits are particularly crucial in this regard. However, current methods rely on expensive tools such as depth cameras and LiDAR. This study proposes a low-cost method based on monocular images to achieve target detection and depth estimation. To improve the detection accuracy of targets, especially small targets, an advanced YOLOv8s detection algorithm is introduced. This approach utilizes the BiFormer block, an attention mechanism for dynamic query-aware sparsity, as the backbone feature extractor. It also adds a small-target-detection layer in the Neck and employs EIoU Loss as the loss function. Furthermore, a fused depth estimation method is proposed, which incorporates high-resolution, low-resolution, and local high-frequency depth estimation to obtain depth information with both high-frequency details and low-frequency structure. Finally, the spatial 3D coordinates of the fruit are obtained by fusing the planar coordinates and depth information. The experimental results with citrus as the target result in an improved YOLOv8s network mAP of 88.45% and a recognition accuracy of 94.7%. The recognition of citrus in a natural environment was improved by 2.7% compared to the original model. In the detection range of 30 cm~60 cm, the depth-estimation results (MAE, RSME) are 0.53 and 0.53. In the illumination intensity range of 1000 lx to 5000 lx, the average depth estimation results (MAE, RSME) are 0.49 and 0.64. In the simulated fruit-picking scenario, the success rates of grasping at 30 cm and 45 cm were 80.6% and 85.1%, respectively. The method has the advantage of high-resolution depth estimation without constraints of camera parameters and fruit size that monocular geometric and binocular localization do not have, providing a feasible and low-cost localization method for fruit automation equipment.

Infrastructure-Based Vehicle Localization through Camera Calibration for I2V Communication Warning.

In recent years, the research on object detection and tracking is becoming important for the development of advanced driving assistance systems (ADASs) and connected autonomous vehicles (CAVs) aiming to improve safety for all road users involved. Intersections, especially in urban scenarios, represent the portion of the road where the most relevant accidents take place; therefore, this work proposes an I2V warning system able to detect and track vehicles occupying the intersection and representing an obstacle for other incoming vehicles. This work presents a localization algorithm based on image detection and tracking by a single camera installed on a roadside unit (RSU). The vehicle position in the global reference frame is obtained thanks to a sequence of linear transformations utilizing intrinsic camera parameters, camera height, and pitch angle to obtain the vehicle's distance from the camera and, thus, its global latitude and longitude. The study brings an experimental analysis of both the localization accuracy, with an average error of 0.62 m, and detection reliability in terms of false positive (1.9%) and missed detection (3.6%) rates.

AdaptMVSNet: Efficient Multi-View Stereo with adaptive convolution and attention fusion

Multi-View Stereo (MVS) is a crucial technique for reconstructing the geometric structure of a scene, given the known camera parameters. Previous deep learning-based MVS methods have mainly focused on improving the reconstruction quality but overlooked the running efficiency during the actual algorithm deployment. For example, deformable convolutions have been introduced to improve the accuracy of the reconstruction results further, however, its inability for parallel optimization caused low inference speed. In this paper, we propose AdaptMVSNet which is device-friendly and reconstruction-efficient, while preserving the original results. To this end, adaptive convolution is introduced to significantly improve the efficiency in speed and metrics compared to current methods. In addition, an attention fusion module is proposed to blend features from adaptive convolution and the feature pyramid network. Our experiments demonstrate that our proposed approach achieves state-of-the-art performance and is almost 2× faster than the recent fastest MVS method. We will release our source code.

Feature selection based on the self-calibration of binocular camera extrinsic parameters

The accuracy of feature-based vision algorithms, including the self-calibration of binocular camera extrinsic parameters used in autonomous driving environment perception techniques relies heavily on the quality of the features extracted from the images. This study investigates the influence of the depth distance between objects and the camera, the feature points in different object regions, and the feature points in dynamic object regions on the self-calibration of binocular camera extrinsic parameters. To achieve this, the study first filters out different types of objects in the image through semantic segmentation. Then, it identifies the areas of dynamic objects and extracts the feature points in the static object region for the self-calibration of binocular camera extrinsic parameters. By calculating the baseline error of the binocular camera and the row alignment error of the matching feature points, this study evaluates the influence of feature points in dynamic object regions, feature points in different object regions, and feature points at different distances on the self-calibration algorithm. The experimental results demonstrate that feature points at static objects close to the camera are beneficial for the self-calibration of extrinsic parameters of binocular camera.

A low-cost 3D reconstruction and measurement system based on structure-from-motion (SFM) and multi-view stereo (MVS) for sewer pipelines

Traditional image-based detection methods for pipelines lack quantification information (e.g., depth, area, and perimeter of defects) despite the high accuracy. Furthermore, existing three-dimensional (3D) reconstruction methods based on laser and depth cameras are expensive and not maneuverable. To remedy these problems, a simple and novel measurement system for sewer pipeline potholes based on low-cost 3D reconstruction is proposed. First, a sparse reconstruction method based on structure-from-motion (SFM) is proposed to estimate the camera parameters and reconstruct the sparse point cloud from multi-view images that are easily acquired. Second, a dense reconstruction method based on multi-view stereo (MVS) is proposed to generate the depth maps and dense 3D points that can provide a stereo display of pipelines. Third, an automatic segmentation and measurement method based on cylinder fitting and projection for pipeline potholes is proposed. The measurement information of potholes is obtained by projection, triangulation and boundary search. Furthermore, a metric calibration method is proposed to convert the voxel size to the actual size. Comparison experiments show that the average errors of maximum depths, mean depths, areas and perimeters between the predicted and real values are 9.48 %, 13.16 %, 6.70 % and 14.01 %, respectively. Furthermore, the measurement method is robust to the point density of the potholes. The whole proposed system only requires several overlapping images taken from ordinary cameras, which is a low-cost and accurate way for 3D reconstruction of pipelines and measurement of defects.

Visual Servoing of Rigid-Link Flexible-Joint Manipulators in the Presence of Unknown Camera Parameters and Boundary Output

Exact position control of the flexible-joint manipulator (FJM) is a challenging task due to the manipulator’s nonlinearity and underactuated characteristic. For the three-dimensional (3-D) rigid-link FJM, the image-based visual servoing (IBVS) approach with link-position output constraint and unknown camera parameters is investigated in this article. The controller is designed to deal with three problems. First, the visual servoing control law is divided into two domains based on the singular perturbation method, where the control input for the fast subsystem is developed to damp out the flexible joint’s vibration. Second, the adaptive updating law for estimating the unknown camera parameters is presented in the slow subsystem. Third, to guarantee that the rigid-link position keep in set constraints, the controller for the slow subsystem is designed by introducing a barrier Lyapunov function. According to the Lyapunov theorem of stability, the proposed controller for the FJM is theoretically proved to be asymptotically stable. Several numerical simulation experiments are provided to illustrate that the presented control scheme is effective.

REALITY-ENHANCED SYNTHETIC IMAGE TRAINING DATASET FOR COMPUTER VISION CONSTRUCTION MONITORING

Recent growth in the availability of aerial and ground-level images from construction sites has created a surge in computer vision-based techniques for construction monitoring applications. To address the need for large volumes of visual data with ground truth for training the underlying machine learning models, the application of synthetic data, particularly from 3D/4D BIM, has gained traction in the research community. However, the existing domain gap between real and synthetic images, such as rendering noise from repeated material texture patterns, randomized positional camera parameters that collect the ground truth of BIM elements, and the poorly visibility of BIM elements to the camera viewpoints within these environments, has negatively impacted the potential use of these synthetic datasets as scale. To address these limitations, this paper presents a new synthetic image generation pipeline that optimizes and integrates camera extrinsic parameters with a novel synthetic image appearance enhancement technique to generate high volume of quality synthetic data with respective ground-truth. The use of synthetic datasets for training progress monitoring models is validated through several real-data segmentation cases that incorporate: 1) the automatic collection of synthetic images and ground-truth annotations from high-LoD BIM model disciplines, 2) optimization of positional camera parameters using element visibility metrics, and 3) the enhancement of realism of synthetic images using a patch-based generative approach. The benefits and the current limitations to automated progress monitoring, as well as robotic path automation and optimization for progress motoring real data collection are discussed.

Extrinsic Calibration of Visual and Inertial Sensors for the Autonomous Vehicle

The extrinsic calibration of visual-inertial sensors has always been a complex problem for autonomous vehicles because the vehicle usually moves on a plane, rendering the calibration problem degenerate. Even if the vehicle drives on a slope, many great works are ineffective due to insufficient excitation. To address these challenges, we propose a method for calibrating the extrinsic parameters of camera and IMU sensors for vehicles, which involves two key contributions. The first is the introduction of a virtual vehicle reference frame, acting as a bridge to calibrate rotation in degenerate case. This involves formulating two least-squares problems to estimate the 3D relative rotation of the camera-IMU. Secondly, we propose an approach for estimating camera-IMU translation when a vehicle is navigating on a slope. This is done by solving a linear system, and these values are then refined using an algorithm that takes into account the gravitational magnitude. Extensive experimental results show that our proposed method effectively estimates rotation in planar motion and translation in sloping motion. Moreover, it achieves competitive accuracy and consistency when compared to state-of-the-art approaches.

Illumination-guided inverse rendering benchmark: Learning real objects with few cameras

The realm of 3D computer vision and graphics has experienced exponential growth recently, enabling the creation of realistic virtual environments and digital representations of real-world objects. Central to this progression are 3D reconstruction methods that facilitate the virtualization of shape, color, and surface details of real objects. Current methods predominantly employ neural scene representations, which despite their efficacy, grapple with limitations such as necessitating a high number of captured images and the complexity of transforming these representations into explicit geometric forms.An alternative strategy that has gained traction is the deployment of methods such as physically-based differentiable rendering (PBDR) and inverse rendering. These approaches require fewer viewpoints, yield explicit format results, and ensure a smoother transition to other representation methods. To meaningfully assess the performance of different 3D reconstruction methods, it is imperative to utilize benchmark scenes for comparison.Despite the existence of standard objects and scenes within the literature, there is a noticeable deficiency in real-world benchmark data that concurrently captures camera, illumination, and scene parameters — all critical to high-fidelity 3D reconstructions using PBDR and inverse rendering-based methods. In this research, we introduce a methodology for capturing real-world scenes as virtual scenes, integrating illumination parameters alongside camera and scene parameters to enhance the veracity of virtual representations. In addition, we introduce a set of ten real-world scenes, along with their virtual counterparts, designed as benchmarks. These benchmarks encompass a fundamental variety of geometric constructs, including convex, concave, plain, and mixed surfaces. Additionally, we demonstrate the 3D reconstruction results of state-of-the-art 3D reconstruction methods employing PBDR in real-world scenes, using both established methodologies and our proposed one.

A novel robot hand-eye calibration method to enhance calibration accuracy based on the POE model

Hand-eye calibration is crucial for vision-based robots using open-loop visual control. Based on the product of exponentials model, this study introduces a novel hand-eye calibration algorithm to improve calibration accuracy. As opposed to the traditional method of solving the AX = XB type in the hand-eye calibration using the marker, we transform the solution into an AX = B type without requiring accurate camera parameters. Firstly, the nominal marker pose in the robot base is described by the nominal kinematic parameters of the robot, camera, and end-effector-to-camera. The actual marker pose is obtained by physically contacting the marker with the probe. Subsequently, the error parameter of the hand-eye pose is defined as linear to the error between the nominal and actual marker poses and can be solved by an iterative least-squares algorithm. Finally, the calibrated hand-eye pose is acquired by multiplying its nominal pose and the exponential mapping of the error parameter. The simulations and real experiments illustrate the effectiveness and stability of the proposed algorithm. Noteworthy, the mean position accuracy of the vision-based robot is improved nearly 20 times after calibration. Furthermore, the comparison experiment demonstrates that the proposed algorithm is preferable for improving hand-eye calibration accuracy.