Mosaicking Research Articles

Seamless Panoramic Image Stitching Based onInvariant Feature Detector and Image Blending

Seamless Panoramic Image Stitching Based onInvariant Feature Detector and Image Blending

Harnessing multimodal data fusion to advance accurate identification of fish feeding intensity

Accurately identifying the fish feeding intensity plays a vital role in aquaculture. While traditional methods are limited by single modality (e.g., water quality, vision, audio), they often lack comprehensive representation, leading to low identification accuracy. In contrast, the multimodal fusion methods leverage the fusion of features from different modalities to obtain richer target features, thereby significantly enhancing the performance of fish feeding intensity assessment (FFIA). In this work a multimodal dataset called MRS-FFIA was introduced. The MRS-FFIA dataset consists of 7611 labelled audio, video and acoustic dataset, and divided the dataset into four different feeding intensity (strong, medium, weak, and none). To address the limitations of single modality methods, a Multimodal Fusion of Fish Feeding Intensity fusion (MFFFI) model was proposed. The MFFFI model is first extracting deep features from three modal data audio (Mel), video (RGB), Acoustic (SI). Then, image stitching techniques are employed to fuse these extracted features. Finally, the fused features are passed through a classifier to obtain the results. The test results show that the accuracy of the fused multimodal information is 99.26%, which improves the accuracy by 12.80%, 13.77%, and 2.86%, respectively, compared to the best results for single-modality (audio, video and acoustic dataset). This result demonstrates that the method proposed in this paper is better at classifying the feeding intensity of fish and can achieve higher accuracy. In addition, compared with the mainstream single-modality approach, the model improves 1.5%–10.8% in accuracy, and the lightweight effect is more obvious. Based on the multimodal fusion method, the feeding decision can be optimised effectively, which provides technical support for the development of intelligent feeding systems.

RecStitchNet: Learning to stitch images with rectangular boundaries

Irregular boundaries in image stitching naturally occur due to freely moving cameras. To deal with this problem, existing methods focus on optimizing mesh warping to make boundaries regular using the traditional explicit solution. However, previous methods always depend on hand-crafted features (e.g., keypoints and line segments). Thus, failures often happen in overlapping regions without distinctive features. In this paper, we address this problem by proposing RecStitchNet, a reasonable and effective network for image stitching with rectangular boundaries. Considering that both stitching and imposing rectangularity are non-trivial tasks in the learning-based framework, we propose a three-step progressive learning based strategy, which not only simplifies this task, but gradually achieves a good balance between stitching and imposing rectangularity. In the first step, we perform initial stitching by a pre-trained state-of-the-art image stitching model, to produce initially warped stitching results without considering the boundary constraint. Then, we use a regression network with a comprehensive objective regarding mesh, perception, and shape to further encourage the stitched meshes to have rectangular boundaries with high content fidelity. Finally, we propose an unsupervised instance-wise optimization strategy to refine the stitched meshes iteratively, which can effectively improve the stitching results in terms of feature alignment, as well as boundary and structure preservation. Due to the lack of stitching datasets and the difficulty of label generation, we propose to generate a stitching dataset with rectangular stitched images as pseudo-ground-truth labels, and the performance upper bound induced from the it can be broken by our unsupervised refinement. Qualitative and quantitative results and evaluations demonstrate the advantages of our method over the state-of-the-art.

Assessing Intra-Bundle Impregnation in Partially Impregnated Glass Fiber-Reinforced Polypropylene Composites Using a 2D Extended-Field and Multimodal Imaging Approach.

This study evaluates multimodal imaging for characterizing microstructures in partially impregnated thermoplastic matrix composites made of woven glass fiber and polypropylene. The research quantifies the impregnation degree of fiber bundles within composite plates manufactured through a simplified compression resin transfer molding process. For comparison, a reference plate was produced using compression molding of film stacks. An original surface polishing procedure was introduced to minimize surface defects while polishing partially impregnated samples. Extended-field 2D imaging techniques, including polarized light, fluorescence, and scanning electron microscopies, were used to generate images of the same microstructure at fiber-scale resolutions throughout the plate. Post-processing workflows at the macro-scale involved stitching, rigid registration, and pixel classification of FM and SEM images. Meso-scale workflows focused on 0°-oriented fiber bundles extracted from extended-field images to conduct quantitative analyses of glass fiber and porosity area fractions. A one-way ANOVA analysis confirmed the reliability of the statistical data within the 95% confidence interval. Porosity quantification based on the conducted multimodal approach indicated the sensitivity of the impregnation degree according to the layer distance from the pool of melted polypropylene in the context of simplified-CRTM. The findings underscore the potential of multimodal imaging for quantitative analysis in composite material production.

Correction Method for Perspective Distortions of Pipeline Images

It is common to find severe perspective distortion in a pipeline’s image in medium-diameter pipeline defect detection by the panoramic image unwrapping method, resulting in low-quality image unwrapping and stitching, which is caused by the camera’s optical axis being completely deviated from the pipeline’s center. To solve this problem, a novel correction method for reducing perspective distortion in pipeline images was proposed for pipeline defect detection. Firstly, the method enhances the edges of unevenly illuminated regions within a pipeline to facilitate image segmentation and identify key points necessary for correcting perspective distortion. Then, a six-feature-point extraction method was proposed for a circle target to establish the projection relationship between the extracted feature and mapped points on the reference circle. Finally, a perspective matrix was constructed to complete the perspective transformation correction of the distorted images. The results show that the average correction rate and the average relative error of the proposed correction method can reach 90.85% and 1.31%, respectively. The study innovatively used the enhancement of uneven illumination to find distorted edge information. It proposed an extraction method using a reference circle and six key feature points to build a mapping model. It can provide a novel method which can be used to obtain a superior image for pipeline detection and lay a solid foundation for subsequent high-quality pipeline image stitching.

Automatic Defect Detection Instrument for Spherical Surfaces of Optical Elements

In order to realize automatic surface defect detection for large aperture precision spherical optical elements, an automatic surface defect detection instrument has been designed. The instrument consists of a microscopic imaging system, illumination system, motion scanning system, and a software algorithm system. Firstly, a multi-angle channel illumination source and a coaxial illumination source were designed. Bright and dark field images of surface defects were captured by cooperating with an automatic zoom microscope. Then, algorithms for scanning trajectory planning, image stitching, and intelligent defect recognition were designed to achieve full-aperture surface image acquisition and defect quantification detection. The automated defect detection process of the instrument is summarized and introduced. Finally, the experimental platform was constructed, which can work well for the optical elements with a maximum diameter of 400 mm and a relative aperture R/D value of 1. It takes about 15 min to detect an optical element with a diameter of 200 mm in dark-field imaging mode. As a result, the minimum line width of scratch detectable is 2 μm and the minimum diameter of pitting detectable is 4 μm. Clearly, the instrument can realize the automatic detection of surface defects of spherical optical elements, and has the advantages of a high efficiency, stability, reliability, quantification, and data traceability.

SAR Image Registration: The Combination of Nonlinear Diffusion Filtering, Hessian Features and Edge Points.

Synthetic aperture radar (SAR) image registration is an important process in many applications, such as image stitching and remote sensing surveillance. The registration accuracy is commonly affected by the presence of speckle noise in SAR images. When speckle noise is intense, the number of image features acquired by single-feature-based methods is insufficient. An SAR image registration method that combines nonlinear diffusion filtering, Hessian features and edge points is proposed in this paper to reduce speckle noise and obtain more image features. The proposed method uses the infinite symmetric exponential filter (ISEF) for image pre-processing and nonlinear diffusion filtering for scale-space construction. These measures can remove speckle noise from SAR images while preserving image edges. Hessian features and edge points are also employed as image features to optimize the utilization of feature information. Experiments with different noise levels, geometric transformations and image scenes demonstrate that the proposed method effectively improves the accuracy of SAR image registration compared with the SIFT-OCT, SAR-SIFT, Harris-SIFT, NF-Hessian and KAZE-SAR algorithms.

Image stitching and target perception for Autonomous Underwater Vehicle-collected side-scan sonar images

Image stitching and target perception for Autonomous Underwater Vehicle-collected side-scan sonar images

Stitching method for panoramic nail fold images based on capillary contour enhancement.

Nail fold capillaroscopy is an important means of monitoring human health. Panoramic nail fold images improve the efficiency and accuracy of examinations. However, the acquisition of panoramic nail fold images is seldom studied and the problem manifests of few matching feature points when image stitching is used for such images. Therefore, this paper presents a method for panoramic nail fold image stitching based on vascular contour enhancement, which first solves the problem of few matching feature points by pre-processing the image with contrast-constrained adaptive histogram equalization (CLAHE), bilateral filtering (BF), and sharpening algorithms. The panoramic images of the nail fold blood vessels are then successfully stitched using the fast robust feature (SURF), fast library of approximate nearest neighbors (FLANN) and random sample agreement (RANSAC) algorithms. The experimental results show that the panoramic image stitched by this paper's algorithm has a field of view width of 7.43 mm, which improves the efficiency and accuracy of diagnosis.

AI-based Precision Enhancement for Change Detection in Drone/UAV Videos

The rapid advancement in drone/UAV technology has led to their extensive use in various domains including surveillance, agriculture, and environmental monitoring. Change detection in UAV videos is crucial for identifying significant alterations in the observed areas. However, challenges such as varying scales, resolutions, and orientations of video frames pose significant hurdles. This paper addresses these challenges by leveraging the SIFT algorithm for panoramic stitching and change detection, followed by the application of the YOLOv8 model for high-precision anomaly detection

A Novel High-Fidelity Simulation for Finishing Operations: Hybrid Image Mosaic and Wavelet Decomposition.

In finishing simulations, achieving accurate results can be challenging due to the minimal amount of material removal and the limited measurement range of surface micro-topography instruments. To overcome these limitations, a novel high-fidelity modeling method combining image mosaic and wavelet decomposition technologies is proposed in this paper. We achieve the stitching of narrow field and high pixel micro morphology images through four steps: image feature extraction, overlapped feature matching, feature fusion, and stitching effect evaluation. On this basis, the wavelet decomposition method is employed to separate detection signals based on their respective frequencies, allowing the establishment of a datum plane and a roughness surface. The point cloud model undergoes a transformation into a continuous geometric model via the Poisson reconstruction algorithm. In the case study, four sample images of an aluminum alloy sheet after barrel finishing were collected using the ZeGage Plus optical profiler. Each image has an actual size of 834.37 μm × 834.37 μm. Subsequently, a comparison was carried out between the physical and simulation experiments. The results clearly indicate that the proposed method has the potential to enhance the accuracy of the finishing simulation by over 30%. The error between the resulting model and the actual surface of the part can be controlled within 1 μm.

Advanced image stitching method and evaluation for underwater structure inspection utilizing planar array cameras

The inspection of underwater structural health is crucial for comprehensive bridge health assessments. In underwater structure imaging, traditional methods include single-camera and binocular camera inspection. However, due to water turbidity and long working distances with a small field-of-view, obtaining clear and high-quality detection images with these methods takes much work. To address this problem, this paper presents a method for planar array image stitching based on Harris corner point extraction, utilizing the advantages of planar array cameras characterized by short working distances and wide field-of-view. The core contribution of this paper is the introduction of an innovative image sequence stitching algorithm utilizing Harris corner point extraction and the combination of the first proposed planar array cameras with the image sequence stitching algorithm, which solves the problem of long distance and small field-of-view during the underwater inspection. The image stitching method involves calibrating camera parameters with a checkerboard and stitching underwater images from planar array cameras to reveal underwater structural features. Furthermore, five quantitative evaluation metrics and the method for calculating the field-of-view loss rate are presented to evaluate and analyze the stitched images. A series of experiments were performed on concrete surfaces, aquatic and underwater, with a total field-of-view of the underwater image after stitching of 358.86 mm × 319.24 mm at a working distance of 160 mm. Five evaluation methods were used to quantitatively evaluate the quality of the stitched images and calculate the field-of-view loss rate of the images. The results indicate that the proposed method improves the ability to inspect underwater. The stitched images achieve notable metrics: an entropy of approximately 6.7, an average gradient of about 1.7, a spatial frequency of around 3.5, an edge strength of about 17, mutual information of approximately 1.2, and a field-of-view loss rate of <0.1, facilitating more effective underwater structure inspection.

SC-AOF: A Sliding Camera and Asymmetric Optical-Flow-Based Blending Method for Image Stitching.

Parallax processing and structure preservation have long been important and challenging tasks in image stitching. In this paper, an image stitching method based on sliding camera to eliminate perspective deformation and asymmetric optical flow to solve parallax is proposed. By maintaining the viewpoint of two input images in the mosaic non-overlapping area and creating a virtual camera by interpolation in the overlapping area, the viewpoint is gradually transformed from one to another so as to complete the smooth transition of the two image viewpoints and reduce perspective deformation. Two coarsely aligned warped images are generated with the help of a global projection plane. After that, the optical flow propagation and gradient descent method are used to quickly calculate the bidirectional asymmetric optical flow between the two warped images, and the optical-flow-based method is used to further align the two warped images to reduce parallax. In the image blending, the softmax function and registration error are used to adjust the width of the blending area, further eliminating ghosting and reducing parallax. Finally, by comparing our method with APAP, AANAP, SPHP, SPW, TFT, and REW, it has been proven that our method can not only effectively solve perspective deformation, but also gives more natural transitions between images. At the same time, our method can robustly reduce local misalignment in various scenarios, with higher structural similarity index. A scoring method combining subjective and objective evaluations of perspective deformation, local alignment and runtime is defined and used to rate all methods, where our method ranks first.

Triangulated Irregular Network based Seamline Determination for Fast Image Stitching of Multiple UAV Images

Abstract. To maintain the efficiency of UAV (unmanned aerial vehicle) remote sensing, rapid image stitching is essential to make multiple UAV images into a seamless mosaic image. Relief displacement introduces variations in object appearance for each image, causing mismatch errors at mosaic seamlines. Traditional approaches involve orthorectifying images using DSMs (digital surface models). While these approaches allow for accurate image stitching, they do not cope with the advantages of UAVs due to their time consumption. In contrast, fast image stitching techniques that do not use orthorectification are well suited for UAV image processing. Related researches have attempted to optimize seamlines to eliminate the errors caused by relief displacement without the use of DSMs. We propose to utilize a TIN (triangular irregular network) of tiepoints to effectively eliminate errors caused by relief displacement while maintaining the fast speed of image stitching. In this study, a TIN is constructed based image tiepoints whose ground coordinates have been obtained through bundle adjustment. The edges of the TIN are used to generate seamlines for image stitching, and the facets of the TIN are used to select minimal images for image stitching and to optimize seamlines. Image stitching results of our proposed method had small error of 1–2 pixels and the processing time of less than 10 minutes for 97 UAV images. This study showed that the proposed method could stitch multiple images while maintaining stable quality using only geometric clues of a TIN of tiepoints.

Advanced Image Stitching Method for Dual-Sensor Inspection.

Efficient image stitching plays a vital role in the Non-Destructive Evaluation (NDE) of infrastructures. An essential challenge in the NDE of infrastructures is precisely visualizing defects within large structures. The existing literature predominantly relies on high-resolution close-distance images to detect surface or subsurface defects. While the automatic detection of all defect types represents a significant advancement, understanding the location and continuity of defects is imperative. It is worth noting that some defects may be too small to capture from a considerable distance. Consequently, multiple image sequences are captured and processed using image stitching techniques. Additionally, visible and infrared data fusion strategies prove essential for acquiring comprehensive information to detect defects across vast structures. Hence, there is a need for an effective image stitching method appropriate for infrared and visible images of structures and industrial assets, facilitating enhanced visualization and automated inspection for structural maintenance. This paper proposes an advanced image stitching method appropriate for dual-sensor inspections. The proposed image stitching technique employs self-supervised feature detection to enhance the quality and quantity of feature detection. Subsequently, a graph neural network is employed for robust feature matching. Ultimately, the proposed method results in image stitching that effectively eliminates perspective distortion in both infrared and visible images, a prerequisite for subsequent multi-modal fusion strategies. Our results substantially enhance the visualization capabilities for infrastructure inspection. Comparative analysis with popular state-of-the-art methods confirms the effectiveness of the proposed approach.

Fast and robust feature-based stitching algorithm for microscopic images

The limited field of view of high-resolution microscopic images hinders the study of biological samples in a single shot. Stitching of microscope images (tiles) captured by the whole-slide imaging (WSI) technique solves this problem. However, stitching is challenging due to the repetitive textures of tissues, the non-informative background part of the slide, and the large number of tiles that impact performance and computational time. To address these challenges, we proposed the Fast and Robust Microscopic Image Stitching (FRMIS) algorithm, which relies on pairwise and global alignment. The speeded up robust features (SURF) were extracted and matched within a small part of the overlapping region to compute the transformation and align two neighboring tiles. In cases where the transformation could not be computed due to an insufficient number of matched features, features were extracted from the entire overlapping region. This enhances the efficiency of the algorithm since most of the computational load is related to pairwise registration and reduces misalignment that may occur by matching duplicated features in tiles with repetitive textures. Then, global alignment was achieved by constructing a weighted graph where the weight of each edge is determined by the normalized inverse of the number of matched features between two tiles. FRMIS has been evaluated on experimental and synthetic datasets from different modalities with different numbers of tiles and overlaps, demonstrating faster stitching time compared to existing algorithms such as the Microscopy Image Stitching Tool (MIST) toolbox. FRMIS outperforms MIST by 481% for bright-field, 259% for phase-contrast, and 282% for fluorescence modalities, while also being robust to uneven illumination.

Automatic image alignment and stitching for ultrasound-based robotic inspection of complex geometry components

Robotic inspection of complex geometry components using immersion ultrasonic techniques is relevant for many ultrasonic Non-Destructive Testing (NDT) applications in different industries. To image large component, ultrasonic probe is manipulated around the component immersed in water using a robotic guided system to create a tiled image of the entire component. Due to the mechanical vibrations of the high-speed robotic arm, the probe position coordinates provided by the robotic system are not precise enough to ensure an accurate reconstruction (stitching) of a composite image from all individual images. In this work, we propose a stitching method specifically designed to create a single 2D image of the surface of an inspected component from a stack of Total Focusing Method (TFM) images. This stitching method uses Iterative Closest Point (ICP) registration to estimate a 2D rigid transformation between two consecutive 2D images, by maximizing the overlap between the surface geometries extracted from each image. The capabilities of this imaging technique are illustrated by various simulated and experimental results carried out in a water tank. Significant improvements in surface image quality, leading to accurate surface reconstruction, are shown for vertical vibrations with displacements of more than two operating wavelengths and for rotational vibrations with deviation angles of less than 1°. The results also show that the resolving power of the ICP algorithm decreases for strong rotational vibrations.

360° Panorama Stitching Method with Depth Information: Enhancing Image Quality and Stitching Accuracy

360° Panorama Stitching Method with Depth Information: Enhancing Image Quality and Stitching Accuracy

Semantic Aware Stitching for Panorama.

The most critical aspect of panorama generation is maintaining local semantic consistency. Objects may be projected from different depths in the captured image. When warping the image to a unified canvas, pixels at the semantic boundaries of the different views are significantly misaligned. We propose two lightweight strategies to address this challenge efficiently. First, the original image is segmented as superpixels rather than regular grids to preserve the structure of each cell. We propose effective cost functions to generate the warp matrix for each superpixel. The warp matrix varies progressively for smooth projection, which contributes to a more faithful reconstruction of object structures. Second, to deal with artifacts introduced by stitching, we use a seam line method tailored to superpixels. The algorithm takes into account the feature similarity of neighborhood superpixels, including color difference, structure and entropy. We also consider the semantic information to avoid semantic misalignment. The optimal solution constrained by the cost functions is obtained under a graph model. The resulting stitched images exhibit improved naturalness. Extensive testing on common panorama stitching datasets is performed on the algorithm. Experimental results show that the proposed algorithm effectively mitigates artifacts, preserves the completeness of semantics and produces panoramic images with a subjective quality that is superior to that of alternative methods.

Internal thread defect detection system based on multi-vision.

In the realm of industrial inspection, the precise assessment of internal thread quality is crucial for ensuring mechanical integrity and safety. However, challenges such as limited internal space, inadequate lighting, and complex geometry significantly hinder high-precision inspection. In this study, we propose an innovative automated internal thread detection scheme based on machine vision, aimed at addressing the time-consuming and inefficient issues of traditional manual inspection methods. Compared with other existing technologies, this research significantly improves the speed of internal thread image acquisition through the optimization of lighting and image capturing devices. To effectively tackle the challenge of image stitching for complex thread textures, an internal thread image stitching technique based on a cylindrical model is proposed, generating a full-view thread image. The use of the YOLOv8 model for precise defect localization in threads enhances the accuracy and efficiency of detection. This system provides an efficient and intuitive artificial intelligence solution for detecting surface defects on geometric bodies in confined spaces.