Camera Parameter Estimation Research Articles

Seamless and Aligned Texture Optimization for 3D Reconstruction

AbstractRestoring the appearance of the model is a crucial step for achieving realistic 3D reconstruction. High‐fidelity textures can also conceal some geometric defects. Since the estimated camera parameters and reconstructed geometry usually contain errors, subsequent texture mapping often suffers from undesirable visual artifacts such as blurring, ghosting, and visual seams. In particular, significant misalignment between the reconstructed model and the registered images will lead to texturing the mesh with inconsistent image regions. However, eliminating various artifacts to generate high‐quality textures remains a challenge. In this paper, we address this issue by designing a texture optimization method to generate seamless and aligned textures for 3D reconstruction. The main idea is to detect misalignment regions between images and geometry and exclude them from texture mapping. To handle the texture holes caused by these excluded regions, a cross‐patch texture hole‐filling method is proposed, which can also synthesize plausible textures for invisible faces. Moreover, for better stitching of the textures from different views, an improved camera pose optimization is present by introducing color adjustment and boundary point sampling. Experimental results show that the proposed method can eliminate the artifacts caused by inaccurate input data robustly and produce high‐quality texture results compared with state‐of‐the‐art methods.

CF-NeRF: Camera Parameter Free Neural Radiance Fields with Incremental Learning

Neural Radiance Fields have demonstrated impressive performance in novel view synthesis. However, NeRF and most of its variants still rely on traditional complex pipelines to provide extrinsic and intrinsic camera parameters, such as COLMAP. Recent works, like NeRFmm, BARF, and L2G-NeRF, directly treat camera parameters as learnable and estimate them through differential volume rendering. However, these methods work for forward-looking scenes with slight motions and fail to tackle the rotation scenario in practice. To overcome this limitation, we propose a novel camera parameter free neural radiance field (CF-NeRF), which incrementally reconstructs 3D representations and recovers the camera parameters inspired by incremental structure from motion. Given a sequence of images, CF-NeRF estimates camera parameters of images one by one and reconstructs the scene through initialization, implicit localization, and implicit optimization. To evaluate our method, we use a challenging real-world dataset, NeRFBuster, which provides 12 scenes under complex trajectories. Results demonstrate that CF-NeRF is robust to rotation and achieves state-of-the-art results without providing prior information and constraints.

Research on Image Stitching Technology for Road Surface Cracks

Image stitching refers to the combination of multiple images with a certain over lappingarea rate acquired by the image acquisition device into one image, which has a full picture of each image. Image stitching technology is an important part in the field of digital image processing. With the increasing requirements for image quality and the limited resolution of ordinary cameras, it is impossible to obtain large angle of view and clear image at the same time. In this paper, the key technologies involved in pavement crack image stitching are intensively studied, including image feature point detection and matching, internal and external camera parameter estimation and optimization, and image fusion. The work done in this paper and the main innovations include: (1)Two main feature detection and description algorithms, SIFT algorithm and ORBalgorithm, are studied. In view of the fact that the SIFT algorithm has redundancy when the Gaussian pyramid is established, this paper appropriately reduces the number of groups and maintains the number and stability of feature point detection. Combined with the advantages and disadvantages of the improved SIFT algorithm and ORB algorithm, and the performance of the two algorithms in the detection of pavement cracks, the improvedSIF'T algorithm is used as the feature point detection algorithm, and rBRlEF algorithm for describing the feature points in the ORB algorithm is used as the feature point description algorithm. The description algorithm is called the BSIF'T algorithm. Experiments show that the robustness of BSIFT algorithm is similar to that of SIFT algorithm, which is better than ORB algorithm and faster than SIF'T algorithm. (2)Image fiusion algorithms are studied, including best seam-line search and image fusion. Aiming at the ghost phenomenon that may occur in overlapping regions, this paper proposes a best seam-line search algorithm based on distance transform.

Solving photogrammetric cold cases using AI-based image matching: New potential for monitoring the past with historical aerial images

With the ongoing digitization in archives, an increasing number of historical data becomes available for research. This includes historical aerial images which provide detailed information about the depicted area. Among the applications enabled by these images are change detection of land use, land cover, glaciers, and coastal environments as well as the observation of land degradation, and natural hazards. Studying the depicted areas and occurring 3D deformations requires the generation of a digital surface model (DSM) which is usually obtained via photogrammetric Structure-from-Motion (SfM). However, conventional SfM workflows often fail in registering historical aerial images due to their radiometric characteristics introduced by digitization, original image quality, or vast temporal changes between epochs. We demonstrate that the feature matching step in the Structure from Motion (SfM) pipeline is particularly crucial. To address this issue, we apply the two synergetic neural network methods SuperGlue and DISK, improving feature matching for historical aerial images. This requires several modifications to enable rotational invariance and leveraging the high resolution of aerial images. In contrast to other studies our workflow does not require any prior information such as DSMs, flight height, focal lengths, or scan resolution which are often no more extent in archives. It is shown that our methods using adapted parameter settings are even able to deal with quasi texture-less images. This enables the simultaneous processing of various kind of mono-temporal and multi-temporal data handled in a single workflow from data preparation over feature matching through to camera parameter estimation and the generation of a sparse point cloud. It outperforms conventional strategies in the number of correct feature matches, number of registered images and calculated 3D points and allows the generation of multi-temporal DSMs with high quality.With the flexibility of the method, it enables the automatic processing of formerly unusable or only to be interactively processed data, e.g. aerial images where the flight route is unknown, or with difficult radiometric properties. This makes it possible to go back even further in time, where the data quality usually decreases, and enables a holistic monitoring and comparison of environments of high interest. The code is made publicly available at https://github.com/tudipffmgt/HAI-SFM.

ShrimpSeg: a local-global structure for the mantis shrimp point cloud segmentation network with contextual reasoning.

To quantify the architecture and select the ideal ideotype, it is vital to accurately measure the dimension of each part of the mantis shrimp. Point clouds have become increasingly popular in recent years as an efficient solution. However, the current manual measurement is labor intensive and costly and has high uncertainty. Automatic organ point cloud segmentation is a prerequisite and core step for phenotypic measurements of mantis shrimps. Nevertheless, little work focuses on mantis shrimp point cloud segmentation. To fill this gap, this paper develops a framework for automated organ segmentation of mantis shrimps from multiview stereo (MVS) point clouds. First, a Transformer-based MVS architecture is applied to generate dense point clouds from a set of calibrated phone images and estimated camera parameters. Next, an improved point cloud segmentation (named ShrimpSeg) that exploits both local and global features based on contextual information is proposed for organ segmentation of mantis shrimps. According to the evaluation results, the per-class intersection over union of organ-level segmentation is 82.4%. Comprehensive experiments demonstrate the effectiveness of ShrimpSeg, outperforming other commonly used segmentation methods. This work may be helpful for improving shrimp phenotyping and intelligent aquaculture at the level of production-ready.

Learning Camera Parameters With Weighted Edge Attention From Single-View Images

This paper presents a novel Deep Learning (DL) model that estimates camera parameters, including camera rotations, field of view, and distortion parameter, from single-view images. The classical approach often analyzes geometric cues such as vanishing points, but is constrained only when geometric cues exist in images. To alleviate such constraints, we use DL, and employ implicit geometric cues, which can reflect the inter-image changes of camera parameters and be observed more frequently in images. Our geometric cues are inspired by two important intuitions: 1) geometric appearance changes caused by camera parameters are the most prominent in object edges; 2) spatially consistent objects (in size and shape) better reflect the inter-image changes of camera parameters. To realize our approach, we propose a weighted edge-attention mechanism that assigns higher weights onto the edges of spatially consistent objects. Our experiments prove that our edge-driven geometric emphasis significantly improves the estimation accuracy of the camera parameters than the existing DL-based approaches.

Automatic orientation of historical terrestrial images in mountainous terrain using the visible horizon

Historical terrestrial images are the only visual sources documenting alpine environments shortly after the end of the Little Ice Age. Despite their unique value, they are largely unused for quantifying environmental changes because of the difficult and time-consuming estimation of the unknown camera parameters. For most images large parts of the captured scenery have vastly changed over time, making automatic feature point matching infeasible. In contrast, the visible image horizon seems to remain stable over time and hence, appears to be a suitable feature for image orientation. Since the focal length is unknown for historical terrestrial images, existing methods, focusing solely on estimating the exterior orientation of recent imagery, can not be applied. Accordingly, it was investigated if the horizon is suitable to estimate both the interior and exterior orientation of historical terrestrial images, with an accuracy comparable to manually oriented images. In a first step, the whole horizon was used to approximate the unknown camera parameters, reducing the potential search space. In the subsequent spatial resection these approximations were further refined using salient points along the horizon. We evaluated our approach using 204 manually oriented reference images. With the proposed method the accuracy of the estimated exterior orientation could be significantly improved compared to previous works. Additionally, the unknown focal length was estimated within 5% of the true focal length for 75% of the images. As historical terrestrial images are commonly used for monoplotting, the accuracy for 2400 manually selected checkpoints was evaluated. This analysis showed that for 63% of the images the same accuracy as with manually oriented images was achieved. For additional 22% the estimated camera parameters were still accurate enough to serve as initial estimates for a subsequent manual orientation. In 15% of the images our method completely failed. Due to the vastly changing scenery and oblique viewing geometry, finding the initial camera parameters, in our experience, is often the most challenging and time consuming step during manual orientation of historical images. Hence, in 85% of the images this initial step can be replaced with our method, leading to a significantly reduced effort for orienting whole collections of historical terrestrial images.

Identification of Location and Camera Parameters for Public Live Streaming Web Cameras

Public live streaming web cameras are quite common now and widely used by drivers for qualitative analysis of traffic conditions. At the same time, they can be a valuable source of quantitative information on transport flows and speed for the development of urban traffic models. However, to obtain reliable data from raw video streams, it is necessary to preprocess them, considering the camera location and parameters without direct access to the camera. Here we suggest a procedure for estimating camera parameters, which allows us to determine pixel coordinates for a point cloud in the camera’s view field and transform them into metric data. They are used with advanced moving object detection and tracking for measurements.

In-orbit geometric calibration of multi-linear array optical remote sensing satellites with tie constraints.

When some sub-images lack ground control points (GCPs) or GCPs are not evenly distributed, the estimated camera parameters are often deviated in in-orbit geometric calibration. In this study, a feasible in-orbit geometric calibration method for multi-linear array optical remote sensing satellites with tie constraints is presented. In the presented method, both GCPs and tie points are employed. With the help of tie constraints provided by tie points, all charge coupled devices (CCDs) are logically connected into a complete CCD. The internal camera parameters of all CCDs can then be simultaneously and precisely estimated, even if sufficient evenly distributed GCPs in some sub-images are unavailable. Three GaoFen-6 images and two ZiYuan3-02 images were tested. Compared with the conventional method, the experimental results showed that the deviations of the estimated camera parameters could be effectively eliminated by the presented method. The average geometric stitching accuracy of the adjacent sub-images of all the tested images were improved from approximately 0.5 pixel to 0.1 pixel. The geometric quality of the stitched images was thereby improved.

ACCURACY ANALYSIS OF ESTIMATED CAMERA ORIENTATION PARAMETERS OF HISTORICAL IMAGES – CASE STUDY OF GEORG-SCHUMANN-BAU IN DRESDEN (GERMANY)

Abstract. Digitalization in archives provides access to an ever-increasing amount of historical photographs. As most of these are not taken by photogrammetric experts it is of interest if these images can be integrated into photogrammetric workflows in order to use them for advanced visualizations. While the pose estimation of historical images can be solved using feature matches retrieved by neural networks it is still unclear how accurate the determined poses are. This contribution tries to provide a reference for unordered historical photo collections with additional contemporary images using the example of the Georg-Schumann-Bau in Dresden, Germany. Therefor, a calibrated camera is used to take images of the building with the idea to enhance the estimated camera parameters of historical images in a combined model. This reference can then be used for comparison with a model created using exclusively historical images. Only half of the estimated poses are below 10 meters compared to the reference which falls behind expectations. Nonetheless, it is possible to use the estimated poses for evaluating aggregations of historical images.

Calibration of focus-adjustable cameras for triangulation sensors by model-based camera parameter estimation

Abstract The working range of triangulation-based measuring systems is often limited by the depth of field of the applied optical system. Approaches such as increasing the depth of field range by reducing the aperture are limited in their effectiveness by a lower light yield and thus require a longer exposure time. Focus adjustable optics enable the depth of field to be modified and thus effect the location and shape of the working range. However, the validity of the model-based camera calibration is limited to the corresponding focus setting, especially with optics that exhibit the effect of focus breathing, i. e. whose focal length change with the focus setting. This paper presents an approach to model the calibration parameters over the entire focus range of a lens using function fitting of the camera parameters and the influence of different configurations on this process is examined.

AUTOMATIC ENRICHMENT OF INDOOR 3D MODELS USING A DEEP LEARNING APPROACH BASED ON SINGLE IMAGES WITH UNKNOWN CAMERA POSES

Abstract. 3D building modeling is a diverse field of research with a multitude of challenges, where data integration is an inherent component. The intensively growing market of BIM-related consumer applications requires methods and algorithms that enable efficient updates of existing 3D models without the need for cost-intensive data capturing and repetitive reconstruction processes. We propose a novel approach for semantic enrichment of existing indoor models by window objects, based on amateur camera RGB images with unknown exterior orientation parameters. The core idea of the approach is the parallel estimation of image camera poses with semantic recognition of target objects and their automatic mapping onto a 3D vector model. The presented solution goes beyond pure texture matching and links deep learning detection techniques with camera pose estimation and 3D reconstruction. To evaluate the performance of our procedure, we compare the estimated camera parameters with reference data, obtaining median values of 13.8 cm for the camera position and 1.1° for its orientation. Furthermore, a quality of 3D mapping is assessed based on the comparison to the reference 3D point cloud. All the windows presented in the data source were detected successfully, with a mean distance between both point sets equal to 3.6 cm. The experimental results prove that the presented approach achieves accurate integration of objects extracted from single images with an input 3D model, allowing for an effective increase of its semantic coverage.

Feature matching for multi-epoch historical aerial images

Historical imagery is characterized by high spatial resolution and stereoscopic acquisitions, providing a valuable resource for recovering 3D land-cover information. Accurate geo-referencing of diachronic historical images by means of self-calibration remains a bottleneck because of the difficulty to find sufficient amount of feature correspondences under evolving landscapes. In this research, we present a fully automatic approach to detecting feature correspondences between historical images taken at different times (i.e., inter-epoch), without auxiliary data required. Based on relative orientations computed within the same epoch (i.e., intra-epoch), we obtain DSMs (Digital Surface Model) and incorporate them in a rough-to-precise matching. The method consists of: (1) an inter-epoch DSMs matching to roughly co-register the orientations and DSMs (i.e, the 3D Helmert transformation), followed by (2) a precise inter-epoch feature matching using the original RGB images. The innate ambiguity of the latter is largely alleviated by narrowing down the search space using the co-registered data. With the inter-epoch feature correspondences, we refine the image orientations and quantitatively evaluate the results (1) with DoD (Difference of DSMs), (2) with ground check points, and (3) by quantifying ground displacement due to an earthquake. We demonstrate that our method: (1) can automatically georeference diachronic historical images; (2) can effectively mitigate systematic errors induced by poorly estimated camera parameters; (3) is robust to drastic scene changes. Compared to the state-of-the-art, our method improves the image georeferencing accuracy by a factor of 2. The proposed methods are implemented in MicMac, a free, open-source photogrammetric software.

UAV Block Geometry Design and Camera Calibration: A Simulation Study.

Acknowledged guidelines and standards such as those formerly governing project planning in analogue aerial photogrammetry are still missing in UAV photogrammetry. The reasons are many, from a great variety of projects goals to the number of parameters involved: camera features, flight plan design, block control and georeferencing options, Structure from Motion settings, etc. Above all, perhaps, stands camera calibration with the alternative between pre- and on-the-job approaches. In this paper we present a Monte Carlo simulation study where the accuracy estimation of camera parameters and tie points’ ground coordinates is evaluated as a function of various project parameters. A set of UAV (Unmanned Aerial Vehicle) synthetic photogrammetric blocks, built by varying terrain shape, surveyed area shape, block control (ground and aerial), strip type (longitudinal, cross and oblique), image observation and control data precision has been synthetically generated, overall considering 144 combinations in on-the-job self-calibration. Bias in ground coordinates (dome effect) due to inaccurate pre-calibration has also been investigated. Under the test scenario, the accuracy gap between different block configurations can be close to an order of magnitude. Oblique imaging is confirmed as key requisite in flat terrain, while ground control density is not. Aerial control by accurate camera station positions is overall more accurate and efficient than GCP in flat terrain.

Deep learning‐based object identification with instance segmentation and pseudo‐LiDAR point cloud for work zone safety

AbstractAutomated object identification in three‐dimensional (3D) space is crucial for work zone safety, such as compliance with construction rules and preventing workplace injuries and deaths. However, it is greatly challenged by some factors like high‐quality detection, high‐quality instance segmentation, few engineering object datasets with masks, and accurate 3D object understanding due to scale variations and limited cues in the 3D world. Traditional hand‐crafted methods suffer from these challenges. Our key insight is to use 2D object detection, instance segmentation and camera vision to compute pseudo‐light detection and ranging (LiDAR) point cloud for 3D object identification. On the one hand, an enhanced feature pyramid network is proposed to extract more fine‐grained object features, and an improved cascade mask R‐CNN is applied to detect bounding boxes and masks for all 2D objects efficiently. Moreover, the AIM dataset for heavy equipment detection is augmented, and a new object class with the bounding box and mask is added. On the other hand, pseudo‐LiDAR point clouds of objects based on bounding boxes and masks are recovered from a monocular image by deep learning, automatic camera parameter estimation, vision‐based method, and space filter. Extensive experiments and analyses show that the new methodology can identify 3D objects and automatically analyze work zone safety. The proposed object detection model has achieved state‐of‐the‐art results on the AIM dataset and 97.2% in mean average precision for the augmented dataset. The collision detection model using pseudo‐LiDAR point cloud has obtained 95.99% in accuracy. The new model will serve as a baseline to support 3D object identification research for other 3D tasks.

Horizon Line Detection in Historical Terrestrial Images in Mountainous Terrain Based on the Region Covariance

Horizon line detection is an important prerequisite for numerous tasks including the automatic estimation of the unknown camera parameters for images taken in mountainous terrain. In contrast to modern images, historical photographs contain no color information and have reduced image quality. In particular, missing color information in combination with high alpine terrain, partly covered with snow or glaciers, poses a challenge for automatic horizon detection. Therefore, a robust and accurate approach for horizon line detection in historical monochrome images in mountainous terrain was developed. For the detection of potential horizon pixels, an edge detector is learned based on the region covariance as texture descriptor. In combination with shortest path search the horizon in monochrome images is accurately detected. We evaluated our approach on 250 selected historical monochrome images in average dating back to 1950. In 85% of the images the horizon was detected with an error less than 10 pixels. In order to further evaluate the performance, an additional dataset consisting of modern color images was used. Our method, using only grayscale information, achieves comparable results with methods based on color information. In comparison with other methods using only grayscale information, accuracy of the detected horizons is significantly improved. Furthermore, the influence of color, choice of neighborhood for the shortest path calculation, and patch size for the calculation of the region covariance were investigated. The results show that both the availability of color information and increasing the patch size for the calculation of the region covariance improve the accuracy of the detected horizons.

SECPNet—secondary encoding network for estimating camera parameters

Camera parameter estimation can be used in visual odometry, robot vision, SLAM, 3D reconstruction and other directions. It is also the main research content of computer vision. Based on the deep learning strategy, we propose a secondary encoding network for camera parameters (SECPNet), which can predict the camera parameters and recover the camera pose according to a single RGB image. Based on the three-dimensional dataset ShapeNet40 (Chang et al. in An information-rich 3D model repository, 2015. arXiv:1512.03012 ), we build a varifocal multi-viewpoint image dataset for camera parameter estimation. Experimental results show that our method has state-of-the-art performance in camera parameter estimation.

Automatic far‐field camera calibration for construction scene analysis

AbstractThe use of cameras for safety monitoring, progress tracking, and site security has grown significantly on construction and civil infrastructure sites over the past decade. Localization of construction resources is a crucial prerequisite for many applications in automated construction management. However, most existing vision‐based methods perform the analysis in the image plane, overlooking the effect of perspective and depth. The manual and labor‐intensive process of traditional calibration techniques, as well as the busy and restrictive construction environment, makes this a challenging task. This study proposes a framework for automatic camera calibration with no manual intervention. The framework utilizes convolutional neural networks for geometrical scene analysis and object detection, which are used to estimate the location of horizon line, vertical vanishing point, as well as objects with known height distributions. This enables automatic estimation of camera parameters and retrieval of scale. The proposed framework is evaluated on images from two major construction projects in Melbourne, Australia. Results show that the proposed method achieves a minimum accuracy of 90% in estimating proximity of points on the ground and can facilitate further development of vision‐based solutions for safety and productivity analysis.

Development of a method for calculation of camera parameters using feature points for which 3D coordinates are known outside and unknown inside the control volume

The purpose of this study was to develop a method for calculation of camera parameters using feature points outside the control volume (known three-dimensional coordinates) and feature points inside the control volume (unknown three-dimensional coordinates, as filmed by multiple cameras). The film process used in this study was as follows: (1) The camera platform position and focal length for properly filming the control volume were selected. (2) Without changing the camera platform position and focal length, the cameras were panned and tilted to film the feature points outside the control volume (number of images = 4). (3) The camera platform joints were adjusted and fixed to the film control volume. Finally, (4) the feature points inside the control volume were filmed by the cameras. From the images obtained, the digitizer coordinates of the feature points inside and outside the control volume were collected. The focal length and camera platform parameters (such as the position, lean, and joint angle) were estimated using hybrid particle swarm optimization and a genetic algorithm to minimize the sum of the reprojective errors of the feature points outside and inside the control volume. The camera parameters at the moment of filming the control volume were calculated using the focal length and camera platform parameters. Three-dimensional coordinates as accuracy verification markers (70 points) inside the control volume were reconstructed using the estimated camera parameters. The RMS errors of the accuracy verification markers in the CCUP method and a DLT method developed in a previous study were 10.5 and 4.4 mm, respectively. These results reveal that the CCUP method can be adapted for collection of three-dimensional coordinates.

Structural-Information-Based Robust Corner Point Extraction for Camera Calibration Under Lens Distortions and Compression Artifacts

Previous camera calibration methods often use a checkerboard to capture images and estimate the camera parameters from the correspondences between images and the checkerboard. The corner points in the checkerboard images are used as useful features for correspondence matching. Therefore, it is essential to precisely find the corner points in the checkerboard images. In many previous works, the corner points are extracted assuming that the checkerboard images are not distorted by its lens. Instead, image blurring and Gaussian noise on the images are usually considered, but other cases are not dealt with. However, the captured checkerboard images are often corrupted by lens distortions and compression artifacts, which leads to performance degradation of corner point extraction. Moreover, the corner points are extracted individually in the previous methods without considering their geometric relations. To better handle the corner point extraction problem under lens distortions, in our corner point extraction optimization, the distorted locations of the pixels on checkerboard images are corrected with the camera parameters, and the structural constraints for checkerboard image grids are then applied under the line-to-line mapping. Also, to robustly find the blurred edges between corner points due to JPEG compression, an edge surface model is newly proposed that models the transitions with over- and under-shoots around the blurred edges. Extensive experimental results show that our method significantly outperforms the state-of-the-art method with average 88.3% and 54.3% reduction in RMSE for corner point reprojection and camera parameter estimation, respectively under compression and lens distortions for synthetic and real data.