Mosaicking Research Articles

Development of techniques for gantryless associated-particle imaging

To move fast-neutron radiography using the associated-particle imaging technique from the laboratory to the field, the development of new analysis techniques is required. In particular, the relative positions of the source and detectors need to be determined when they have been placed by hand, the normalization for a particular source–detector geometry needs to be determined without a measurement in the same geometry with no object present, and accurate image stitching is required when multiple detector positions are necessary to image an object. The present work describes methods that employ transmission neutron data to localize a fast-neutron imaging panel with respect to the neutron source, calculate a normalization for a given source–detector geometry, stitch images together, and describes the required system calibrations. The reported techniques enable in-field neutron radiography for cases in which the source–detector geometry is not well known a priori and where operational constraints preclude a normalization measurement.

An open-source method for producing reliable water temperature maps for ecological applications using non-radiometric sensors

The number of thermal sensors mounted on Unmanned Aerial Vehicles (UAVs) has increased considerably in recent years, with several of those currently available on the market being very attractive due to their low cost and ease of use. UAV platforms coupled with non-radiometric thermal sensors tend to be least expensive, but few studies have evaluated the reliability of data they produce or developed workflows to improve data accuracy. Here we present a complete method for obtaining accurate and precise water temperature data from non-radiometric sensors, including bias correction, calibration, image stitching and wetted area extraction. We illustrate the utility of the method through its application to the assessment of thermal conditions in two tropical river reaches. The whole procedure is implemented in the R language, and we provide a tutorial and customizable functions. Validation tests indicate that our method produces water temperature estimates that are comparable to those from more expensive sensors (mean absolute error <0.5 °C) and so can be used to answer ecological questions with great confidence. The method dramatically increases the accessibility of thermal imaging in ecology by providing accurate data that is cost-effective and not computationally intensive, and for calibration and validation relies on only small and lightweight equipment that can transported and used in remote locations.

RTHEN: Unsupervised deep homography estimation based on dynamic attention for repetitive texture image stitching

Homography estimation is regarded as one of the key challenges in image alignment, where the goal is to estimate the projective transformation between two images on the same plane. Unsupervised learning methods are gradually becoming popular due to their excellent performance and lack of need for labeled data. However, in regional scenes with repeated textures, there may be ambiguity in the correspondence between local features, affecting homography estimation accuracy. This paper proposes a new unsupervised deep homography method RTHEN to solve such problems. In order to effectively obtain repeated texture features, a multi-scale Feature pyramid Siamese network (FPSN) is designed. Specifically, we dynamically allocate the weights of recited texture features through a dynamic attention module and introduce a channel attention module to provide rich contextual information for repeated texture areas. We propose a hard triplet loss function based on overlap constraints to optimize the matching results. At the same time, we collected a repetitive texture image dataset (RTID) for homography estimation training and evaluation. Experimental results show that our method outperforms existing learning methods in repetitive texture scenes and offers competitive performance with state-of-the-art traditional methods.

A survey of feature matching methods

AbstractFeature matching plays a crucial role in computer vision, with applications in visual localization, simultaneous localization and mapping (SLAM), image stitching, and more. It establishes correspondences between sets of feature points from multiple images, enabling various tasks. Over the years, feature matching has witnessed significant development, with an increasing number of methods being applied. However, different methods exhibit different degrees of applicability in different scenarios and requirements due to their different rationales. To cope with these issues, a comprehensive analysis and comparison of matching methods are essential. Existing reviews often lack coverage of deep learning models and focus more on feature detection and description, neglecting the matching process. This survey investigates feature detection, description, and matching techniques within the feature‐based image‐matching pipeline. Representative methods, their mechanisms, and application scenarios are also briefly introduced. In addition, comprehensive evaluations of classical and state‐of‐the‐art methods are conducted through extensive experiments on representative datasets. Particularly, matching‐based applications are compared to fully demonstrate the advantages of the methods. Lastly, this survey highlights current problems and development directions in matching methods, serving as a reference for researchers in the field.

WebODM: An Open-Source Alternative to Commercial Image Stitching Software for Uncrewed Aerial Systems

This publication aims to help UAS operators become familiar with advanced data processing tools integrated into the WebODM FOSS. The central aim of this publication is to underscore the process and sequence involved in producing data and outcomes by acquainting users with the fundamental features of WebODM. Written by Sahaj Patel, Jesse Chintanadilok, Brittany Hall-Scharf, Yilin Zhuang, Joseph Strickland, and Aditya Singh, and published by the UF/IFAS Department of Agricultural and Biological Engineering, February 2024.

A self-adaptive parallel image stitching algorithm for unmanned aerial vehicles in edge computing environments

ABSTRACT The overlap between edge computing and unmanned aerial systems enables Unmanned Aerial Vehicles (UAV) to quickly offload image processing tasks onto edge devices, avoiding the transmission of images over long distances. To improve the speed and efficiency of UAV image stitching in an edge computing environment, this paper proposes an adaptive UAV image parallel stitching algorithm in an edge computing environment. The algorithm incorporates both route-based parallel processing and inverted binary tree-based parallel processing, dividing the image stitching task into multiple processes and allocating them to different cores based on the CPU core count, number of flight routes, and number of images, thereby enhancing computational efficiency in edge scenarios. The experimental results indicate that, when the number of flight routes is greater than or equal to the number of CPUs, the adaptive algorithm will employ the more efficient route parallelism. Conversely, when the number of flight routes is less than the number of CPUs, the efficiency of inverted binary tree parallelism is higher. In the same experimental environment and dataset, the adaptive image stitching algorithm demonstrates an efficiency improvement of approximately 2–10 times compared to other algorithms, with no significant degradation in image quality. This demonstrates that in edge environments, the utilization of multi-threaded adaptive route and inverted binary tree-based parallel approaches can effectively harness the computing resources of edge devices, significantly improving the stitching speed of UAV images and providing technical support for rapid real-time monitoring by UAV.

A Novel Framework for Image Matching and Stitching for Moving Car Inspection under Illumination Challenges.

Vehicle exterior inspection is a critical operation for identifying defects and ensuring the overall safety and integrity of vehicles. Visual-based inspection of moving objects, such as vehicles within dynamic environments abounding with reflections, presents significant challenges, especially when time and accuracy are of paramount importance. Conventional exterior inspections of vehicles require substantial labor, which is both costly and prone to errors. Recent advancements in deep learning have reduced labor work by enabling the use of segmentation algorithms for defect detection and description based on simple RGB camera acquisitions. Nonetheless, these processes struggle with issues of image orientation leading to difficulties in accurately differentiating between detected defects. This results in numerous false positives and additional labor effort. Estimating image poses enables precise localization of vehicle damages within a unified 3D reference system, following initial detections in the 2D imagery. A primary challenge in this field is the extraction of distinctive features and the establishment of accurate correspondences between them, a task that typical image matching techniques struggle to address for highly reflective moving objects. In this study, we introduce an innovative end-to-end pipeline tailored for efficient image matching and stitching, specifically addressing the challenges posed by moving objects in static uncalibrated camera setups. Extracting features from moving objects with strong reflections presents significant difficulties, beyond the capabilities of current image matching algorithms. To tackle this, we introduce a novel filtering scheme that can be applied to every image matching process, provided that the input features are sufficient. A critical aspect of this module involves the exclusion of points located in the background, effectively distinguishing them from points that pertain to the vehicle itself. This is essential for accurate feature extraction and subsequent analysis. Finally, we generate a high-quality image mosaic by employing a series of sequential stereo-rectified pairs.

Panoramic visual system for spherical mobile robots

Abstract Aimed at the challenges of wide-angle mobile robot visual perception for diverse field applications, we present the spherical robot visual system that uses a 360° field of view (FOV) for realizing real-time object detection. The spherical robot image acquisition system model is developed with optimal parameters, including camera spacing, camera axis angle, and the distance of the target image plane. Two 180 $^{\circ}$ -wide panoramic FOVs, front and rear view, are formed using four on-board cameras. The speed of the SURF algorithm is increased for feature extraction and matching. For seamless fusion of the images, an improved fade-in and fade-out algorithm is used, which not only improves the seam quality but also improves object detection performance. The speed of the dynamic image stitching is significantly enhanced by using a cache-based sequential image fusion method. On top of the acquired panoramic wide FOVs, the YOLO algorithm is used for real-time object detection. The panoramic visual system for the spherical robot is then tested in real time, which outputs panoramic views of the scene at an average frame rate of 21.69 fps and panoramic views with object detection at an average of 15.39 fps.

Image Stitching via Augmentation Selection and Comprehensive Optimization of Homography Models

Image Stitching via Augmentation Selection and Comprehensive Optimization of Homography Models

Panoramic image stitching technology and its application in the field of autonomous driving

Image processing technology is an indispensable part of the development of many fields, in which panoramic image stitching technology is one of the most widely used technologies in image processing, which is not only applied in the frontier fields of technology such as unmanned aerial vehicles and machine vision detection, but is also closely related to people's lives, for example, the panoramic camera shooting function of the smart phone. The driverless car is also a widely concerned field, its development means the liberation of manpower, the realization and improvement of self-driving car technology can alleviate traffic congestion, the use of transportation for people without the ability to drive and many other social problems. This paper will firstly introduce the panoramic collage technology, including its principle, system architecture and application prospect. Secondly, it focuses on the application of panoramic collage technology in the field of self-driving cars, which is divided into two aspects: sensor hardware and collocation algorithm, and analyzes the applicability of different algorithmic techniques in this field based on the demand characteristics of self-driving cars. The research in this paper points out the optimization direction of panoramic collage technology required by self-driving cars, which will be of great value to the development and application of panoramic collage technology in the field of self-driving cars.

An Anti-Noise-Designed Residual Phase Unwrapping Neural Network for Digital Speckle Pattern Interferometry

DSPI (Digital Speckle Pattern Interferometry) is a non-destructive optical measurement technique that obtains phase information of an object through phase unwrapping. Traditional phase unwrapping algorithms depend on the quality of the images, which demands preprocessing such as filtering and denoising. Moreover, the unwrapping time is highly influenced by the size of the images. In this study, we proposed a new deep learning-based phase unwrapping algorithm combining the residual network and U-Net network. Additionally, we incorporated an improved SSIM function as the loss function based on camera characteristics. The experimental results demonstrated that the proposed method achieved higher quality in highly noisy phase unwrapping maps compared to traditional algorithms, with SSIM values consistently above 0.98. In addition, we applied image stitching to the network to process maps of various sizes and the unwrapping time remained around 1 s even for larger images. In conclusion, our proposed network is able to achieve efficient and accurate phase unwrapping.

PE-RASP: range image stitching of photon-efficient imaging through reconstruction, alignment, stitching integration network based on intensity image priors.

Single photon imaging integrates advanced single photon detection technology with Laser Radar (LiDAR) technology, offering heightened sensitivity and precise time measurement. This approach finds extensive applications in biological imaging, remote sensing, and non-visual field imaging. Nevertheless, current single photon LiDAR systems encounter challenges such as low spatial resolution and a limited field of view in their intensity and range images due to constraints in the imaging detector hardware. To overcome these challenges, this study introduces a novel deep learning image stitching algorithm tailored for single photon imaging. Leveraging the robust feature extraction capabilities of neural networks and the richer feature information present in intensity images, the algorithm stitches range images based on intensity image priors. This innovative approach significantly enhances the spatial resolution and imaging range of single photon LiDAR systems. Simulation and experimental results demonstrate the effectiveness of the proposed method in generating high-quality stitched single-photon intensity images, and the range images exhibit comparable high quality when stitched with prior information from the intensity images.

SatellStitch: Satellite Imagery-Assisted UAV Image Seamless Stitching for Emergency Response without GCP and GNSS

Rapidly stitching unmanned aerial vehicle (UAV) imagery to produce high-resolution fast-stitch maps is key to UAV emergency mapping. However, common problems such as gaps and ghosting in image stitching remain challenging and directly affect the visual interpretation value of the imagery product. Inspired by the data characteristics of high-precision satellite images with rich access and geographic coordinates, a seamless stitching method is proposed for emergency response without the support of ground control points (CGPs) and global navigation satellite systems (GNSS). This method aims to eliminate stitching traces and solve the problem of stitching error accumulation. Firstly, satellite images are introduced to support image alignment and geographic coordinate acquisition simultaneously using matching relationships. Then a dynamic contour point set is constructed to locate the stitching region and adaptively extract the fused region of interest (FROI). Finally, the gradient weight cost map of the FROI image is computed and the Laplacian pyramid fusion rule is improved to achieve seamless production of the fast-stitch image map with geolocation information. Experimental results indicate that the method is well adapted to two representative sets of UAV images. Compared with the Laplacian pyramid fusion algorithm, the peak signal-to-noise ratio (PSNR) of the image stitching results can be improved by 31.73% on average, and the mutual information (MI) can be improved by 19.98% on average. With no reliance on CGPs or GNSS support, fast-stitch image maps are more robust in harsh environments, making them ideal for emergency mapping and security applications.

3-D Contrast-Enhanced Fusion Ultrasound for Accurate Volume Assessment of Vessel Lumen and Plaque in Carotid Artery Disease as Compared With Computed Tomography Angiography

ObjectiveThree-dimensional contrast-enhanced fusion ultrasound (3D-CEFUS) of atherosclerotic carotid arteries provides spatial visualisation of the vessel lumen, creating a lumenography. As in 3D computed tomography angiography (3D-CTA), 3D-CEFUS outlines the contrast-filled lumen. Plaque and vessel contours are distinguished in 3D-CEFUS, allowing plaque volume quantification as a valid estimate of carotid plaque burden. 3D-CEFUS is unproven in inter-modality studies, vindicating the assessment of 3D-CEFUS applicability and comparing 3D-CEFUS and 3D-CTA lumenography as a proof-of-concept study. MethodsUsing an ultrasound system with magnetic tracking, a linear array transducer, and SonoVue contrast agent generated 3D-CEFUS acquisitions by spatial stitching of serial 2D images. From 3D-CEFUS and 3D-CTA imaging, the atherosclerotic carotid arteries were reconstructed with lumenography in an offline software programme for lumen and plaque volume quantification. Bland-Altman analysis was used for inter-image-modality agreement. ResultsThe study included 39 carotid arteries. Mean lumen and plaque volume in 3D-CEFUS were 0.63 cm3 (SD 0.26) and 0.62 cm3 (SD 0.26), respectively. Lumen volume differences between 3D-CEFUS and 3D-CTA were non-significant, with mean difference of 0.01 cm3 (SD 0.02, p = .26) and limits of agreement range (LoA-range) of ±0.11 cm3. Mean plaque volume difference was -0.12 cm3 (SD 0.19, p = .006) with LoA-range of ±0.39 cm3. ConclusionStrong agreement in lumenography between 3D-CEFUS and 3D-CTA was found. The inter-image-modality difference in plaque volumes was substantial due to challenging vessel wall definition in 3D-CTA. 3D-CEFUS is viable in quantifying carotid plaque volume burden and can potentially monitor plaque development over time.

Temporal structured illumination and vision-transformer enables large field-of-view binary snapshot ptychography.

Ptychography, a widely used computational imaging method, generates images by processing coherent interference patterns scattered from an object of interest. In order to capture scenes with large field-of-view (FoV) and high spatial resolution simultaneously in a single shot, we propose a temporal-compressive structured-light Ptychography system. A novel three-step reconstruction algorithm composed of multi-frame spectra reconstruction, phase retrieval, and multi-frame image stitching is developed, where we employ the emerging Transformer-based network in the first step. Experimental results demonstrate that our system can expand the FoV by 20× without losing spatial resolution. Our results offer huge potential for enabling lensless imaging of molecules with large FoV as well as high spatial-temporal resolutions. We also notice that due to the loss of low-intensity information caused by the compressed sensing process, our method so far is only applicable to binary targets.

VEMstitch: an algorithm for fully automatic image stitching of volume electron microscopy.

As software and hardware have developed, so has the scale of research into volume electron microscopy (vEM), leading to ever-increasing resolution. Usually, data collection is followed by image stitching: the same area is subjected to high-resolution imaging with a certain overlap, and then the images are stitched together to achieve ultrastructure with large scale and high resolution simultaneously. However, there is currently no perfect method for image stitching, especially when the global feature distribution of the sample is uneven and the feature points of the overlap area cannot be matched accurately, which results in ghosting of the fusion area. We have developed a novel algorithm called vEMstitch to solve these problems, aiming for seamless and clear stitching of high-resolution images. In vEMstitch, the image transformation model is constructed as a combination of global rigid and local elastic transformation using weighted pixel displacement fields. Specific local geometric constraints and feature reextraction strategies are incorporated to ensure that the transformation model accurately and completely reflects the characteristics of biological distortions. To demonstrate the applicability of vEMstitch, we conducted thorough testing on simulated datasets involving different transformation combinations, consistently showing promising performance. Furthermore, in real data sample experiments, vEMstitch successfully gives clear ultrastructure in the stitching region, reaffirming the effectiveness of the algorithm. vEMstitch serves as a valuable tool for large-field and high-resolution image stitching. The clear stitched regions facilitate better visualization and identification in vEM analysis. The source code is available at https://github.com/HeracleBT/vEMstitch.

DunHuangStitch: Unsupervised Deep Image Stitching of Dunhuang Murals.

The digital construction of cultural heritage promotes communication and sharing of digital cultural resources across time and space. Digital storage serves as the foundation for the digital construction of cultural artifacts. In the digital storage of Dunhuang murals, image stitching plays a critical role in restoring the complete image of the cave murals. Traditional image stitching methods are constrained by the detection accuracy of feature points and are not fit for stitching low-texture murals. Despite deep learning-based image stitching methods, parallax misalignment and ghosting are still prevalent issues. For this reason, we perform the first Dunhuang mural stitching based on deep learning in this paper. This is in response to the need for digitizing and storing Dunhuang murals. Two mural stitching datasets are constructed, and we design a progressive regression image alignment network and a feature differential reconstruction soft-coded seam stitching network. We also introduce a soft-coded seam quality evaluation method. The algorithm presented in this paper achieves state-of-the-art alignment and stitching performance in the mural stitching task through unsupervised learning with a smaller number of model parameters, which provides technical support for the digitization and preservation of Dunhuang murals. The codes and models will be available at https://github.com/MmelodYy/DunHuangStitch.

Unmanned Aerial Vehicle Based Power Pole Localization Using Image Stitching

When facing natural disasters, power transmission and distribution towers are prone to tilting, collapsing, and other situations, which can easily lead to line faults and subsequently cause power outage incidents. In order to quickly and accurately locate abnormal towers and restore power supply promptly, the power maintenance department typically employs unmanned aerial vehicles (UAVs) for post-disaster inspections of power line towers. Traditional methods for UAV ground target localization mainly rely on the UAV’s own position, attitude, and onboard sensor information to indirectly calculate the target’s position. This method is influenced by the Global Positioning System (GPS) and Inertial Navigation System (INS), relying on high-precision sensors. To achieve precise positioning of power line towers with simplified equipment on UAVs, this paper proposes a visual passive localization method based on ground control points. Addressing the sparse distribution of ground control points, a solution based on image stitching is introduced to overcome difficulties in localization caused by insufficient ground control points.

Multispectral Image Stitching via Global-Aware Quadrature Pyramid Regression.

Image stitching is a critical task in panorama perception that involves combining images captured from different viewing positions to reconstruct a wider field-of-view (FOV) image. Existing visible image stitching methods suffer from performance drops under severe conditions since environmental factors can easily impair visible images. In contrast, infrared images possess greater penetrating ability and are less affected by environmental factors. Therefore, we propose an infrared and visible image-based multispectral image stitching method to achieve all-weather, broad FOV scene perception. Specifically, based on two pairs of infrared and visible images, we employ the salient structural information from the infrared images and the textual details from the visible images to infer the correspondences within different modality-specific features. For this purpose, a multiscale progressive mechanism coupled with quadrature correlation is exploited to improve regression in different modalities. Exploiting the complementary properties, accurate and credible homography can be obtained by integrating the deformation parameters of the two modalities to compensate for the missing modality-specific information. A global-aware guided reconstruction module is established to generate an informative and broad scene, wherein the attentive features of different viewpoints are introduced to fuse the source images with a more seamless and comprehensive appearance. We construct a high-quality infrared and visible stitching dataset for evaluation, including real-world and synthetic sets. The qualitative and quantitative results demonstrate that the proposed method outperforms the intuitive cascaded fusion-stitching procedure, achieving more robust and credible panorama generation. Code and dataset are available at https://github.com/Jzy2017/MSGA.

GRiD: Guided Refinement for Detector-Free Multimodal Image Matching.

Multimodal image matching is essential in image stitching, image fusion, change detection, and land cover mapping. However, the severe nonlinear radiometric distortion (NRD) and geometric distortions in multimodal images severely limit the accuracy of multimodal image matching, posing significant challenges to existing methods. Additionally, detector-based methods are prone to feature point offset issues in regions with substantial modal differences, which also hinder the subsequent fine registration and fusion of images. To address these challenges, we propose a guided refinement for detector-free multimodal image matching (GRiD) method, which weakens feature point offset issues by establishing pixel-level correspondences and utilizes reference points to guide and correct matches affected by NRD and geometric distortions. Specifically, we first introduce a detector-free framework to alleviate the feature point offset problem by directly finding corresponding pixels between images. Subsequently, to tackle NRD and geometric distortion in multimodal images, we design a guided correction module that establishes robust reference points (RPs) to guide the search for corresponding pixels in regions with significant modality differences. Moreover, to enhance RPs reliability, we incorporate a phase congruency module during the RPs confirmation stage to concentrate RPs around image edge structures. Finally, we perform finer localization on highly correlated corresponding pixels to obtain the optimized matches. We conduct extensive experiments on four multimodal image datasets to validate the effectiveness of the proposed approach. Experimental results demonstrate that our method can achieve sufficient and robust matches across various modality images and effectively suppress the feature point offset problem.