Mosaicking Research Articles

Advancing Electric Engineering Education through Immersive Virtual Reality: Deep Learning and Evolutionary Algorithms for Image Stitching and Rectification in Virtual Lab Environments

Virtual Reality (VR) technology has emerged as a transformative tool in education, offering immersive and interactive experiences that enhance learning outcomes. This paper delves into the application of image stitching and rectification techniques to create a VR lab environment, specifically tailored for electrical engineering education. The importance of VR technology in education is explored, highlighting its role in promoting active learning and providing experiential learning opportunities. The primary emphasis of this Paper lies in the smooth incorporation of image stitching algorithms for the creation of panoramic perspectives, along with the implementation of rectification techniques to correct irregular borders within the stitched images. By utilizing Convolutional Neural Networks (CNNs) and Genetic Algorithms (GAs), the proposed approach optimizes the rectification process, resulting in visually cohesive representations. Demonstrating the utilization of the VR lab across a range of situations, such as examining power transfer and creating control panels for water pumps in irrigation initiatives, the immersive setting enables students to delve into intricate systems. The performance of the proposed method was evaluated using various metrics, including mean squared error, peak signal to noise ratio (PSNR), structural similarity index (SSIM), and Fréchet inception distance (FID). the combination of deep learning algorithm specifically (CNN) and optimization algorithm specifically (Genetic algorithm (GA)) led to an increase in the accuracy of the rectified images where the average PSNR reached 23.98, SSIM was 0.8066, and FID was 18.72. Regarding the users’ opinion about the generated environment by stitching and rectifying images, participants demonstrated consistent positive sentiments, with mean scores ranging from 3.65 to 4.03, all above the scale midpoint, and moderate variability indicated by standard deviation values ranging from 1.070 to 1.251, suggesting general favorability with some variation in responses. This experience empowers the users to gain insights and cultivate essential problemsolving abilities at a heightened level. Collaborative learning is facilitated, enabling students to engage in collaborative projects regardless of their physical location. Through the synthesis of image processing techniques and VR technology, this research contributes to the enrichment of educational experiences and the advancement of electrical engineering education.

Enhancing Situational Awareness with VAS-Compass Net for the Recognition of Directional Vehicle Alert Sounds

People with hearing impairments often face increased risks related to traffic accidents due to their reduced ability to perceive surrounding sounds. Given the cost and usage limitations of traditional hearing aids and cochlear implants, this study aims to develop a sound alert assistance system (SAAS) to enhance situational awareness and improve travel safety for people with hearing impairments. We proposed the VAS-Compass Net (Vehicle Alert Sound–Compass Net), which integrates three lightweight convolutional neural networks: EfficientNet-lite0, MobileNetV3-Small, and GhostNet. Through employing a fuzzy ranking ensemble technique, our proposed model can identify different categories of vehicle alert sounds and directions of sound sources on an edge computing device. The experimental dataset consisted of images derived from the sounds of approaching police cars, ambulances, fire trucks, and car horns from various directions. The audio signals were converted into spectrogram images and Mel-frequency cepstral coefficient images, and they were fused into a complete image using image stitching techniques. We successfully deployed our proposed model on a Raspberry Pi 5 microcomputer, paired with a customized smartwatch to realize an SAAS. Our experimental results demonstrated that VAS-Compass Net achieved an accuracy of 84.38% based on server-based computing and an accuracy of 83.01% based on edge computing. Our proposed SAAS has the potential to significantly enhance the situational awareness, alertness, and safety of people with hearing impairments on the road.

Image Stitching of Low-Resolution Retinography Using Fundus Blur Filter and Homography Convolutional Neural Network

Great advances in stitching high-quality retinal images have been made in recent years. On the other hand, very few studies have been carried out on low-resolution retinal imaging. This work investigates the challenges of low-resolution retinal images obtained by the D-EYE smartphone-based fundus camera. The proposed method uses homography estimation to register and stitch low-quality retinal images into a cohesive mosaic. First, a Siamese neural network extracts features from a pair of images, after which the correlation of their feature maps is computed. This correlation map is fed through four independent CNNs to estimate the homography parameters, each specializing in different corner coordinates. Our model was trained on a synthetic dataset generated from the Microsoft Common Objects in Context (MSCOCO) dataset; this work added an important data augmentation phase to improve the quality of the model. Then, the same is evaluated on the FIRE retina and D-EYE datasets for performance measurement using the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM). The obtained results are promising: the average PSNR was 26.14 dB, with an SSIM of 0.96 on the D-EYE dataset. Compared to the method that uses a single neural network for homography calculations, our approach improves the PSNR by 7.96 dB and achieves a 7.86% higher SSIM score.

Phenotyping of Panicle Number and Shape in Rice Breeding Materials Based on Unmanned Aerial Vehicle Imagery.

The number of panicles per unit area (PNpA) is one of the key factors contributing to the grain yield of rice crops. Accurate PNpA quantification is vital for breeding high-yield rice cultivars. Previous studies were based on proximal sensing with fixed observation platforms or unmanned aerial vehicles (UAVs). The near-canopy images produced in these studies suffer from inefficiency and complex image processing pipelines that require manual image cropping and annotation. This study aims to develop an automated, high-throughput UAV imagery-based approach for field plot segmentation and panicle number quantification, along with a novel classification method for different panicle types, enhancing PNpA quantification at the plot level. RGB images of the rice canopy were efficiently captured at an altitude of 15 m, followed by image stitching and plot boundary recognition via a mask region-based convolutional neural network (Mask R-CNN). The images were then segmented into plot-scale subgraphs, which were categorized into 3 growth stages. The panicle vision transformer (Panicle-ViT), which integrates a multipath vision transformer and replaces the Mask R-CNN backbone, accurately detects panicles. Additionally, the Res2Net50 architecture classified panicle types with 4 angles of 0°, 15°, 45°, and 90°. The results confirm that the performance of Plot-Seg is comparable to that of manual segmentation. Panicle-ViT outperforms the traditional Mask R-CNN across all the datasets, with the average precision at 50% intersection over union (AP50) improved by 3.5% to 20.5%. The PNpA quantification for the full dataset achieved superior performance, with a coefficient of determination (R 2) of 0.73 and a root mean square error (RMSE) of 28.3, and the overall panicle classification accuracy reached 94.8%. The proposed approach enhances operational efficiency and automates the process from plot cropping to PNpA prediction, which is promising for accelerating the selection of desired traits in rice breeding.

Recent Advances in Intraoral Scanners.

Intraoral scanners (IOSs) have emerged as a cornerstone technology in digital dentistry. This article examines the recent advancements and multifaceted applications of IOSs, highlighting their benefits in patient care and addressing their current limitations. The IOS market has seen a competitive surge. Modern IOSs, featuring continuous image capture and advanced software for seamless image stitching, have made the scanning process more efficient. Patient comfort with IOS procedures is favorable, mitigating the discomfort associated with conventional impression taking. There has been a shift toward open data interfaces, notably enhancing interoperability. However, the integration of IOSs into large dental institutions is slow, facing challenges such as compatibility with existing health record systems and extensive data storage management. IOSs now extend beyond their use in computer-aided design and manufacturing, with software solutions transforming them into platforms for diagnostics, patient communication, and treatment planning. Several IOSs are equipped with tools for caries detection, employing fluorescence technologies or near-infrared imaging to identify carious lesions. IOSs facilitate quantitative monitoring of tooth wear and soft-tissue dimensions. For precise tooth segmentation in intraoral scans, essential for orthodontic applications, developers are leveraging innovative deep neural network-based approaches. The clinical performance of restorations fabricated based on intraoral scans has proven to be comparable to those obtained using conventional impressions, substantiating the reliability of IOSs in restorative dentistry. In oral and maxillofacial surgery, IOSs enhance airway safety during impression taking and aid in treating conditions such as cleft lip and palate, among other congenital craniofacial disorders, across diverse age groups. While IOSs have improved various aspects of dental care, ongoing enhancements in usability, diagnostic accuracy, and image segmentation are crucial to exploit the potential of this technology in optimizing patient care.

Comparative Analysis of Image Stitching Algorithms for Bridge Inspection Imaging Systems

This paper discusses an innovative vehicle-mounted device designed for efficient inspection of bridge undersides, aiming to address the challenges of manual inspections, such as being labor-intensive and inefficient. The device uses high-precision cameras to capture detailed images of bridge bottoms, facilitating a thorough condition assessment. It evaluates three feature point extraction algorithms for image stitching—Scale-Invariant Feature Transform (SIFT), Harris corner detection, and Speeded Up Robust Features (SURF)—to create complete visual representations of bridge undersides. Field tests on a prefabricated I girder bridge demonstrated the device's effectiveness in gathering and stitching images, with the SIFT algorithm performing best for girder bottoms, Harris corner algorithm for web and flange surfaces, and SURF for rapid image processing. This research provides a foundation for the rapid identification of visible defects on bridge superstructures, significantly benefiting bridge maintenance and safety evaluations.

Distinguishing of Histopathological Staging Features of H-E Stained Human cSCC by Microscopical Multispectral Imaging.

Cutaneous squamous cell carcinoma (cSCC) is the second most common malignant skin tumor. Early and precise diagnosis of tumor staging is crucial for long-term outcomes. While pathological diagnosis has traditionally served as the gold standard, the assessment of differentiation levels heavily depends on subjective judgments. Therefore, how to improve the diagnosis accuracy and objectivity of pathologists has become an urgent problem to be solved. We used multispectral imaging (MSI) to enhance tumor classification. The hematoxylin and eosin (H&E) stained cSCC slides were from Shanghai Ruijin Hospital. Scale-invariant feature transform was applied to multispectral images for image stitching, while the adaptive threshold segmentation method and random forest segmentation method were used for image segmentation, respectively. Synthetic pseudo-color images effectively highlight tissue differences. Quantitative analysis confirms significant variation in the nuclear area between normal and cSCC tissues (p < 0.001), supported by an AUC of 1 in ROC analysis. The AUC within cSCC tissues is 0.57. Further study shows higher nuclear atypia in poorly differentiated cSCC tissues compared to well-differentiated cSCC (p < 0.001), also with an AUC of 1. Lastly, well differentiated cSCC tissues show more and larger keratin pearls. These results have shown that combined MSI with imaging processing techniques will improve H&E stained human cSCC diagnosis accuracy, and it will be well utilized to distinguish histopathological staging features.

A UAV Thermal Imaging Format Conversion System and Its Application in Mosaic Surface Microthermal Environment Analysis.

UAV thermal infrared remote sensing technology, with its high flexibility and high temporal and spatial resolution, is crucial for understanding surface microthermal environments. Despite DJI Drones' industry-leading position, the JPG format of their thermal images limits direct image stitching and further analysis, hindering their broad application. To address this, a format conversion system, ThermoSwitcher, was developed for DJI thermal JPG images, and this system was applied to surface microthermal environment analysis, taking two regions with various local zones in Nanjing as the research area. The results showed that ThermoSwitcher can quickly and losslessly convert thermal JPG images to the Geotiff format, which is further convenient for producing image mosaics and for local temperature extraction. The results also indicated significant heterogeneity in the study area's temperature distribution, with high temperatures concentrated on sunlit artificial surfaces, and low temperatures corresponding to building shadows, dense vegetation, and water areas. The temperature distribution and change rates in different local zones were significantly influenced by surface cover type, material thermal properties, vegetation coverage, and building layout. Higher temperature change rates were observed in high-rise building and subway station areas, while lower rates were noted in water and vegetation-covered areas. Additionally, comparing the temperature distribution before and after image stitching revealed that the stitching process affected the temperature uniformity to some extent. The described format conversion system significantly enhances preprocessing efficiency, promoting advancements in drone remote sensing and refined surface microthermal environment research.

Development and Implementation of a Portable Cigarette Filter Holes Recognition Device Based on Low-Power Image Sensor and High-Precision Coaxial Rotary Scanning

During the cigarette production process, the detection of holes in the filters presents an urgent challenge for manufacturers. Utilizing image sensors offers an effective solution to tackle this challenge effectively. While current detection devices mitigate this issue through the use of auxiliary light source technology and sophisticated deep learning image processing, they are often large-scale, stationary, and demand substantial computational resources. To overcome this limitation, a low-power battery-powered portable filter holes detection device is designed based on image processing to facilitate portability and sampling. Additionally, backlight transmission lighting components are employed to guarantee aperture definition after imaging. Furthermore, a filter holes recognition algorithm is proposed based on multi-threshold combination of computer vision (CV). The algorithm forms a plan view of the circumference of the cigarette by image stitching, then performs filter holes detection and filters the results. Experimental results show that operators can easily carry the device for sampling and testing, and can accurately perform filter holes detection, and the results provide valuable insights into the adjustment of system parameters in the cigarette production process. This device significantly aids in enhancing efficiency during the manual inspection processes both in the testing room and at the equipment stations.

Optimizing algorithm for high-precision imaging stitching systems based on spline surfaces

As optical systems continue to advance, non-uniform rational B-spline (NURBS) surfaces increasingly being considered in asymmetric optical systems due to their localized control characteristics. However, the representation of NURBS surfaces has complicated the analysis of these systems, leading to a significant computational burden. To address this challenge, we propose an optimizing algorithm for imaging optical systems based on high-precision ray tracing of NURBS surfaces. This method initiates with getting a knot grid as prior information, in conjunction with the Newton-Raphson algorithm, to obtain high-precision numerical solutions for the intersection of rays with a NURBS surface. Building upon this methodology, we introduce an optimization technique that includes shape evaluation to generate an evaluation function specific to NURBS surfaces. This approach is then applied within a rapid optimization process that accounted for the region of ray influence. Under consistent control point grids and sampling ray conditions, we present an off-axis four-mirror system to showcase that our algorithm has achieved a computational efficiency improvement of approximately 14 times compared to the previous method. This high-precision imaging design based on spline surfaces fulfills the need for efficient and accurate algorithms for NURBS surface applications in various imaging systems, providing guidance for practical applications.

Gastrointestinal image stitching based on improved unsupervised algorithm.

Image stitching is a traditional but challenging computer vision task. The goal is to stitch together multiple images with overlapping areas into a single, natural-looking, high-resolution image without ghosts or seams. This article aims to increase the field of view of gastroenteroscopy and reduce the missed detection rate. To this end, an improved depth framework based on unsupervised panoramic image stitching of the gastrointestinal tract is proposed. In addition, preprocessing for aberration correction of monocular endoscope images is introduced, and a C2f module is added to the image reconstruction network to improve the network's ability to extract features. A comprehensive real image data set, GASE-Dataset, is proposed to establish an evaluation benchmark and training learning framework for unsupervised deep gastrointestinal image splicing. Experimental results show that the MSE, RMSE, PSNR, SSIM and RMSE_SW indicators are improved, while the splicing time remains within an acceptable range. Compared with traditional image stitching methods, the performance of this method is enhanced. In addition, improvements are proposed to address the problems of lack of annotated data, insufficient generalization ability and insufficient comprehensive performance in image stitching schemes based on supervised learning. These improvements provide valuable aids in gastrointestinal examination.

Local-Peak Scale-Invariant Feature Transform for Fast and Random Image Stitching.

Image stitching aims to construct a wide field of view with high spatial resolution, which cannot be achieved in a single exposure. Typically, conventional image stitching techniques, other than deep learning, require complex computation and are thus computationally expensive, especially for stitching large raw images. In this study, inspired by the multiscale feature of fluid turbulence, we developed a fast feature point detection algorithm named local-peak scale-invariant feature transform (LP-SIFT), based on the multiscale local peaks and scale-invariant feature transform method. By combining LP-SIFT and RANSAC in image stitching, the stitching speed can be improved by orders compared with the original SIFT method. Benefiting from the adjustable size of the interrogation window, the LP-SIFT algorithm demonstrates comparable or even less stitching time than the other commonly used algorithms, while achieving comparable or even better stitching results. Nine large images (over 2600 × 1600 pixels), arranged randomly without prior knowledge, can be stitched within 158.94 s. The algorithm is highly practical for applications requiring a wide field of view in diverse application scenes, e.g., terrain mapping, biological analysis, and even criminal investigation.

UAV-based field watermelon detection and counting using YOLOv8s with image panorama stitching and overlap partitioning

Accurate watermelon yield estimation is crucial to the agricultural value chain, as it guides the allocation of agricultural resources as well as facilitates inventory and logistics planning. The conventional method of watermelon yield estimation relies heavily on manual labor, which is both time-consuming and labor-intensive. To address this, this work proposes an algorithmic pipeline that utilizes unmanned aerial vehicle (UAV) videos for detection and counting of watermelons. This pipeline uses You Only Look Once version 8 s (YOLOv8s) with panorama stitching and overlap partitioning, which facilitates the overall number estimation of watermelons in field. The watermelon detection model, based on YOLOv8s and obtained using transfer learning, achieved a detection accuracy of 99.20 %, demonstrating its potential for application in yield estimation. The panorama stitching and overlap partitioning based detection and counting method uses panoramic images as input and effectively mitigates the duplications compared with the video tracking based detection and counting method. The counting accuracy reached over 96.61 %, proving a promising application for yield estimation. The high accuracy demonstrates the feasibility of applying this method for overall yield estimation in large watermelon fields.

Image stitching algorithm for super-resolution localization microscopy combined with fluorescence noise prior.

Super-resolution panoramic pathological imaging provides a powerful tool for biologists to observe the ultrastructure of samples. Localization data can maintain the essential ultrastructural information of biological samples with a small storage space, and also provides a new opportunity for stitching super-resolution images. However, the existing image stitching methods based on localization data cannot accurately calculate the registration offset of sample regions with no or few structural points and thus lead to registration errors. Here, we proposed a stitching framework called PNanoStitcher. The framework fully utilizes the distribution characteristics of the background fluorescence noise in the stitching region and solves the stitching failure in sample regions with no or few structural points. We verified our method using both simulated and experimental datasets, and compared it with existing stitching methods. PNanoStitcher achieved superior stitching results on biological samples with no structural and few structural regions. The study provides an important driving force for the development of super-resolution digital pathology.

Improved Unsupervised Stitching Algorithm for Multiple Environments SuperUDIS.

Large field-of-view images are increasingly used in various environments today, and image stitching technology can make up for the limited field of view caused by hardware design. However, previous methods are constrained in various environments. In this paper, we propose a method that combines the powerful feature extraction capabilities of the Superpoint algorithm and the exact feature matching capabilities of the Lightglue algorithm with the image fusion algorithm of Unsupervised Deep Image Stitching (UDIS). Our proposed method effectively improves the situation where the linear structure is distorted and the resolution is low in the stitching results of the UDIS algorithm. On this basis, we make up for the shortcomings of the UDIS fusion algorithm. For stitching fractures of UDIS in some complex situations, we optimize the loss function of UDIS. We use a second-order differential Laplacian operator to replace the difference in the horizontal and vertical directions to emphasize the continuity of the structural edges during training. Combined with the above improvements, the Super Unsupervised Deep Image Stitching (SuperUDIS) algorithm is finally formed. SuperUDIS has better performance in both qualitative and quantitative evaluations compared to the UDIS algorithm, with the PSNR index increasing by 0.5 on average and the SSIM index increasing by 0.02 on average. Moreover, the proposed method is more robust in complex environments with large color differences or multi-linear structures.

Seamless Panoramic Image Stitching Based onInvariant Feature Detector and Image Blending

Seamless Panoramic Image Stitching Based onInvariant Feature Detector and Image Blending

Harnessing multimodal data fusion to advance accurate identification of fish feeding intensity

Accurately identifying the fish feeding intensity plays a vital role in aquaculture. While traditional methods are limited by single modality (e.g., water quality, vision, audio), they often lack comprehensive representation, leading to low identification accuracy. In contrast, the multimodal fusion methods leverage the fusion of features from different modalities to obtain richer target features, thereby significantly enhancing the performance of fish feeding intensity assessment (FFIA). In this work a multimodal dataset called MRS-FFIA was introduced. The MRS-FFIA dataset consists of 7611 labelled audio, video and acoustic dataset, and divided the dataset into four different feeding intensity (strong, medium, weak, and none). To address the limitations of single modality methods, a Multimodal Fusion of Fish Feeding Intensity fusion (MFFFI) model was proposed. The MFFFI model is first extracting deep features from three modal data audio (Mel), video (RGB), Acoustic (SI). Then, image stitching techniques are employed to fuse these extracted features. Finally, the fused features are passed through a classifier to obtain the results. The test results show that the accuracy of the fused multimodal information is 99.26%, which improves the accuracy by 12.80%, 13.77%, and 2.86%, respectively, compared to the best results for single-modality (audio, video and acoustic dataset). This result demonstrates that the method proposed in this paper is better at classifying the feeding intensity of fish and can achieve higher accuracy. In addition, compared with the mainstream single-modality approach, the model improves 1.5%–10.8% in accuracy, and the lightweight effect is more obvious. Based on the multimodal fusion method, the feeding decision can be optimised effectively, which provides technical support for the development of intelligent feeding systems.

RecStitchNet: Learning to stitch images with rectangular boundaries

Irregular boundaries in image stitching naturally occur due to freely moving cameras. To deal with this problem, existing methods focus on optimizing mesh warping to make boundaries regular using the traditional explicit solution. However, previous methods always depend on hand-crafted features (e.g., keypoints and line segments). Thus, failures often happen in overlapping regions without distinctive features. In this paper, we address this problem by proposing RecStitchNet, a reasonable and effective network for image stitching with rectangular boundaries. Considering that both stitching and imposing rectangularity are non-trivial tasks in the learning-based framework, we propose a three-step progressive learning based strategy, which not only simplifies this task, but gradually achieves a good balance between stitching and imposing rectangularity. In the first step, we perform initial stitching by a pre-trained state-of-the-art image stitching model, to produce initially warped stitching results without considering the boundary constraint. Then, we use a regression network with a comprehensive objective regarding mesh, perception, and shape to further encourage the stitched meshes to have rectangular boundaries with high content fidelity. Finally, we propose an unsupervised instance-wise optimization strategy to refine the stitched meshes iteratively, which can effectively improve the stitching results in terms of feature alignment, as well as boundary and structure preservation. Due to the lack of stitching datasets and the difficulty of label generation, we propose to generate a stitching dataset with rectangular stitched images as pseudo-ground-truth labels, and the performance upper bound induced from the it can be broken by our unsupervised refinement. Qualitative and quantitative results and evaluations demonstrate the advantages of our method over the state-of-the-art.

Assessing Intra-Bundle Impregnation in Partially Impregnated Glass Fiber-Reinforced Polypropylene Composites Using a 2D Extended-Field and Multimodal Imaging Approach.

This study evaluates multimodal imaging for characterizing microstructures in partially impregnated thermoplastic matrix composites made of woven glass fiber and polypropylene. The research quantifies the impregnation degree of fiber bundles within composite plates manufactured through a simplified compression resin transfer molding process. For comparison, a reference plate was produced using compression molding of film stacks. An original surface polishing procedure was introduced to minimize surface defects while polishing partially impregnated samples. Extended-field 2D imaging techniques, including polarized light, fluorescence, and scanning electron microscopies, were used to generate images of the same microstructure at fiber-scale resolutions throughout the plate. Post-processing workflows at the macro-scale involved stitching, rigid registration, and pixel classification of FM and SEM images. Meso-scale workflows focused on 0°-oriented fiber bundles extracted from extended-field images to conduct quantitative analyses of glass fiber and porosity area fractions. A one-way ANOVA analysis confirmed the reliability of the statistical data within the 95% confidence interval. Porosity quantification based on the conducted multimodal approach indicated the sensitivity of the impregnation degree according to the layer distance from the pool of melted polypropylene in the context of simplified-CRTM. The findings underscore the potential of multimodal imaging for quantitative analysis in composite material production.

Correction Method for Perspective Distortions of Pipeline Images

It is common to find severe perspective distortion in a pipeline’s image in medium-diameter pipeline defect detection by the panoramic image unwrapping method, resulting in low-quality image unwrapping and stitching, which is caused by the camera’s optical axis being completely deviated from the pipeline’s center. To solve this problem, a novel correction method for reducing perspective distortion in pipeline images was proposed for pipeline defect detection. Firstly, the method enhances the edges of unevenly illuminated regions within a pipeline to facilitate image segmentation and identify key points necessary for correcting perspective distortion. Then, a six-feature-point extraction method was proposed for a circle target to establish the projection relationship between the extracted feature and mapped points on the reference circle. Finally, a perspective matrix was constructed to complete the perspective transformation correction of the distorted images. The results show that the average correction rate and the average relative error of the proposed correction method can reach 90.85% and 1.31%, respectively. The study innovatively used the enhancement of uneven illumination to find distorted edge information. It proposed an extraction method using a reference circle and six key feature points to build a mapping model. It can provide a novel method which can be used to obtain a superior image for pipeline detection and lay a solid foundation for subsequent high-quality pipeline image stitching.