Multi-sensor Image Research Articles

One respiratory cycle as a minimum time unit for making behavioral decisions in the mammalian olfactory system.

Voluntary behaviors such as sniffing, moving, and eating require decision-making accompanied by intentional respiration. Based on the study of respiration-coherent activity of rodent olfactory networks, we infer that during the inhalation phase of respiration, olfactory cortical areas process environmental odor information and transmit it to the higher multisensory cognitive areas via feedforward pathways to comprehensively evaluate the surrounding situation. We also infer that during the exhalation phase, the higher multisensory areas generate cognitive-signals and transmit them not only to the behavioral output system but also back to the olfactory cortical areas. We presume that the cortical mechanism couples the intentional respiration with the voluntary behaviors. Thus, in one respiratory cycle, the mammalian brain may transmit and process sensory information to cognize and evaluate the multisensory image of the external world, leading to one behavioral decision and one emotional expression. In this perspective article, we propose that one respiratory cycle provides a minimum time unit for decision making during wakefulness.

Integrating temporal-aggregated satellite image with multi-sensor image fusion for seasonal land-cover mapping of Shilansha watershed, rift valley basin of Ethiopia

Accurate land-cover mapping in regions with frequent cloud-cover and rapidly changing agricultural land cover by crop growth cycles cannot be guaranteed by use of single sensor images, or an image from a single-acquisition-date. This study addressed these challenges by applying temporal-aggregation of single sensor image features that is integrated with multi-sensor image fusion. Results of land-cover classification target fallow, growing, and harvest/post-harvest agricultural seasons. Satellite based features used were frequency bands of Sentinel-1 (S1) and Sentinel-2 (S2), including vegetation indices (VIs) and biophysical variables (BPVs). Temporal aggregation improved classification accuracy. The single-acquisition-date S2 image, overall accuracy (OA) ranged from 0.81 to 0.85, increased to 0.86 to 0.87 after temporal-aggregation. Meanwhile, for single-acquisitions of S1, OA ranged from 0.44 to 0.79 increased to 0.6 to 0.86 across respective seasons. Fusing temporally aggregated S1 and S2 image features including VIs and BPVs increased OA up to 0.90. Selecting 11, 8, and 10 out of 18 optimum numbers of features for fallow, growing, and harvest/post-harvest seasons respectively improved OA by 3%, 2%, and 1.86%. PCA fusion of the temporally aggregated best performing feature set enhanced harvest/post-harvest season, fallow, and growing seasons with OA of 0.98, 0.96 and 0.94 respectively. Accuracy was enhanced when selecting different best performing feature set for the three seasons. The study enhanced knowledge of advanced remote sensing for agricultural land cover mapping, with practical implications of land monitoring and management.

Editorial: Multi-sensor imaging and fusion: methods, evaluations, and applications, volume II

Editorial: Multi-sensor imaging and fusion: methods, evaluations, and applications, volume II

HVDFusion: an effective fusion framework based on Hilbert vibration decomposition for multi-focal and multi-sensor images

HVDFusion: an effective fusion framework based on Hilbert vibration decomposition for multi-focal and multi-sensor images

Exploring idyllic healing: developing a multi-sensory image scale for idyllic life and its impact on restorative effects and travel intention

ABSTRACT Under the theme of “idyllic healing”, this article explores the sensory images of idyllic life and their impact on restorative effects and travel intention to reveal the potential for sensory attraction and restorative effects in idyllic tourism destinations. Moreover, research indicates that imagination possesses significant positive emotional and restorative effects. However, empirical validation within the field of tourism is limited, necessitating further scholarly investigation. The article contains two progressive studies. Study 1 constructed an idyllic life multi-sensory image scale containing seven dimensions: vision, audition, haptic sense, taste, olfaction, kinesthesia and atmospheric sense, based on online text data. Study 2 used structural equation modelling to investigate the relationship between sensory dimensions, verified the influence mechanism of sensory images on restorative effects and travel intention, and found that atmospheric sense and restorative effects played an important mediating role. Through the development of the idyllic life multi-sensory image scale, this paper visualises the sensory image of idyllic life and reveals the positive contributions of imagined sensory images associated with an idyllic life to recovery effects and travel intention, underscoring the significant role of imagination, expanding the field of research in cognitive psychology, providing literature support for the integration of mental health and tourism and recreation, and highlighting the potential of rural destinations as healing places. Moreover, the relationship between the sensory dimensions of idyllic life is verified, which enriches the theory of sensory marketing. This study identifies specific sensory elements that enhance restorative experiences and will help to their translation into measurable environmental variables using psychophysics. Building on this foundation, further research into tourism dosage, this approach aims to objectify perceptions and support the scientification and quantification of travel prescriptions. Additionally, the research is expected to facilitate the future quantification of sensory perceptions and establish optimal standards for physical environmental stimuli in rural tourism settings.

Combining multisensor images and social network data to assess the area flooded by a hurricane event.

In this study, multisensor remote sensing datasets were used to characterize the land use and land covers (LULC) flooded by Hurricane Willa which made landfall on October 24, 2018. The landscape characterization was done using an unsupervised K-means algorithm of a cloud-free Sentinel-2 MultiSpectral Instrument (MSI) image, acquired during the dry season before Hurricane Willa. A flood map was derived using the histogram thresholding technique over a Synthetic Aperture Radar (SAR) Sentinel-1 C-band and combined with a flood map derived from a Sentinel-2 MSI image. Both, the Sentinel-1 and Sentinel-2 images were obtained after Willa landfall. While the LULC map reached an accuracy of 92%, validated using data collected during field surveys, the flood map achieved 90% overall accuracy, validated using locations extracted from social network data, that were manually georeferenced. The agriculture class was the dominant land use (about 2,624 km2), followed by deciduous forest (1,591 km2) and sub-perennial forest (1,317 km2). About 1,608 km2 represents the permanent wetlands (mangrove, salt marsh, lagoon and estuaries, and littoral classes), but only 489 km2 of this area belongs to aquatic surfaces (lagoons and estuaries). The flooded area was 1,225 km2, with the agricultural class as the most impacted (735 km2). Our analysis detected the saltmarsh class occupied 541 km2in the LULC map, and around 328 km2 were flooded during Hurricane Willa. Since the water flow receded relatively quickly, obtaining representative imagery to assess the flood event was a challenge. Still, the high overall accuracies obtained in this study allow us to assume that the outputs are reliable and can be used in the implementation of effective strategies for the protection, restoration, and management of wetlands. In addition, they will improve the capacity of local governments and residents of Marismas Nacionales to make informed decisions for the protection of vulnerable areas to the different threats derived from climate change.

Multi-Sensor Image and Range-Based Techniques for the Geometric Documentation and the Photorealistic 3D Modeling of Complex Architectural Monuments.

The selection of the optimal methodology for the 3D geometric documentation of cultural heritage is a subject of high concern in contemporary scientific research. As a matter of fact, it requires a multi-source data acquisition process and the fusion of datasets from different sensors. This paper aims to demonstrate the workflow for the proper implementation and integration of geodetic, photogrammetric and laser scanning techniques so that high-quality photorealistic 3D models and other documentation products can be generated for a complicated, large-dimensional architectural monument and its surroundings. As a case study, we present the monitoring of the Mehmet Bey Mosque, which is a landmark in the city of Serres and a significant remaining sample of the Ottoman architecture in Greece. The surveying campaign was conducted in the context of the 2022-2023 annual workshop of the Interdepartmental Program of Postgraduate Studies "Protection Conservation and Restoration of Cultural Monuments" of the Aristotle University of Thessaloniki, and it served as a geometric background for interdisciplinary cooperation and decision-making on the monument restoration process. The results of our study encourage the fusion of terrestrial laser scanning and photogrammetric datasets for the 3D modeling of the mosque, as they supplement each other as regards geometry and texture.

Change detection on multi-sensor imagery using mixed interleaved group convolutional network

The difference of the spatio-spectral features of multi-sensor image causes big difficulty in change detection because of the difficulties of the feature extraction. Unlike the traditional approaches that mainly relying on manually feature design, the advances of deep learning-based methods in deep feature extraction provide new alternatives for multi-sensor imagery change detection. Specifically, the incorporation of multi-scale information from remote sensing images holds paramount importance in change detection, consistently applied in the design of various deep learning models. This study investigated a change detection approach utilizing a mixed interleaved group convolutional network (MIGCNet) on multi-sensor remote sensing imagery, with a specific focus on fine-grained kernel space and multi-scale feature analysis within convolution operations. The proposed MIGCNet, with parallel branches as the fundamental architecture, can distinguish the change information effectively by the proposed mixed interleaved group convolution (MIGC) module, which combined mixed convolution with interleaved group convolution. Meanwhile, multi-loss supervision is utilized to promote the performance of the proposed MIGCNet. Experimental results demonstrate the outperformance of the MIGCNet to handle change detection with multi-sensor images on urban area. Considering different datasets, the Overall Accuracy and Kappa Coefficient are reaching 0.97 and 80.67%, respectively, and the miss detection rate and the false alarm rate are as low as 0.17 and 0.18, respectively.

Development of Multi Modal Image Fusion Techniques in Night Vision Technology Using Deep Learning

This study is aimed at exploring new development in nighttime image processing based on nighttime photography/night vision. We use various learning methodologies based on neural networks to develop multisensor data processing techniques. The ability of Convolutional Neural Networks (CNN) to enhance image quality and clarity under conditions of low light or at night is the focus of our research. To do this A large database has been created, containing 3200 images both taken from the internet as well as our own. This data is preprocessed, normalized and features expanded using some common methods of data augmentation in order to optimize model performance. We compared our proposed CNN model with existing literature through a series of experiments. Performance was measured using Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Feature Similarity Index (FSIM). The Results are now in. They showed that our model can improve various metrics with higher scores. And they show the real-time monitoring capabilities in urban streets for our model. This research brings multisensor image fusion based on night-adjacent techniques forward from night vision technology and provides insight into practical and potential future directions for using deep learning techniques to improve safety and security in many different environments.

Structure similarity virtual map generation network for optical and SAR image matching

Introduction: Optical and SAR image matching is one of the fields within multi-sensor imaging and fusion. It is crucial for various applications such as disaster response, environmental monitoring, and urban planning, as it enables comprehensive and accurate analysis by combining the visual information of optical images with the penetrating capability of SAR images. However, the differences in imaging mechanisms between optical and SAR images result in significant nonlinear radiation distortion. Especially for SAR images, which are affected by speckle noises, resulting in low resolution and blurry edge structures, making optical and SAR image matching difficult and challenging. The key to successful matching lies in reducing modal differences and extracting similarity information from the images.Method: In light of this, we propose a structure similarity virtual map generation network (SVGNet) to address the task of optical and SAR image matching. The core innovation of this paper is that we take inspiration from the concept of image generation, to handle the predicament of image matching between different modalities. Firstly, we introduce the Attention U-Net as a generator to decouple and characterize optical images. And then, SAR images are consistently converted into optical images with similar textures and structures. At the same time, using the structural similarity (SSIM) to constrain structural spatial information to improve the quality of generated images. Secondly, a conditional generative adversarial network is employed to further guide the image generation process. By combining synthesized SAR images and their corresponding optical images in a dual channel, we can enhance prior information. This combined data is then fed into the discriminator to determine whether the images are true or false, guiding the generator to optimize feature learning. Finally, we employ least squares loss (LSGAN) to stabilize the training of the generative adversarial network.Results and Discussion: Experiments have demonstrated that the SVGNet proposed in this paper is capable of effectively reducing modal differences, and it increases the matching success rate. Compared to direct image matching, using image generation ideas results in a matching accuracy improvement of more than twice.

Applying deep learning to real-time UAV-based forest monitoring: Leveraging multi-sensor imagery for improved results

Rising global fire incidents necessitate effective solutions, with forest surveillance emerging as a crucial strategy. This paper proposes a complete solution using technology that integrates visible and infrared spectrum images through Unmanned Aerial Vehicles (UAVs) for enhanced detection of people and vehicles in forest environments. Unlike existing computer vision models relying on single-sensor imagery, this approach overcomes limitations posed by limited spectrum coverage, particularly addressing challenges in low-light conditions, fog, or smoke. The developed 4-channel model uses both types of images to take advantage of the strengths of each one simultaneously. This article presents the development and implementation of a solution for forest monitoring ranging from the transmission of images captured by a UAV to their analysis with an object detection model without human intervention. This model consists of a new version of the YOLOv5 (You Only Look Once) architecture. After the model analyzes the images, the results can be observed on a web platform on any device, anywhere in the world. For the model training, a dataset with thermal and visible images from the aerial perspective was captured with a UAV. From the development of this proposal, a new 4-channel model was created, presenting a substantial increase in precision and mAP (Mean Average Precision) metrics compared to traditional SOTA (state-of-the-art) models that only make use of red, green, and blue (RGB) images. Allied with the increase in precision, we confirmed the hypothesis that our model would perform better in conditions unfavorable to RGB images, identifying objects in situations with low light and reduced visibility with partial occlusions. With the model’s training using our dataset, we observed a significant increase in the model’s performance for images in the aerial perspective. This study introduces a modular system architecture featuring key modules: multisensor image capture, transmission, processing, analysis, and results presentation. Powered by an innovative object detection deep-learning model, these components collaborate to enable real-time, efficient, and distributed forest monitoring across diverse environments.

Transferring Deep Models for Cloud Detection in Multisensor Images via Weakly Supervised Learning

Transferring Deep Models for Cloud Detection in Multisensor Images via Weakly Supervised Learning

Online Crack Detection of Highly Curved Cylindrical Coils

Many vision-based methods for defect detection have been proposed to replace inefficient manual inspection. However, due to the complex shape and high degree of freedom required to manipulate individual coils, an automatic optical inspection (AOI) method for cylindrical coils remains unsuccessful due to the difficulty of automatically scanning the front, rear, outer, and inner faces of complex spiral-shaped cylindrical coils. This paper presents a turnkey integrated system that can be operated in real time for real coil manufacturers. First, a novel 1-degree-of-freedom rotational progressive system is designed to automatically transport and sort individual spiral coils. Second, a multisensor imaging system is designed to capture images of 360° panoramic views of a cylindrical coil. Then, a robust automatic defect detection algorithm consisting of crack detection and geometric measurement is developed to detect defects in cylindrical coils during manufacturing procedures. The results of a large-scale validation experiment indicate that the proposed AOI system is robust and is practically immune to variations in surface normal vectors, occlusion, and defocus blur, and it has a high sensitivity and specificity of 99.75% and 94.2%, respectively. This validation experiment proved that the software and hardware designed in this study can implement defect detection of spiral coils.

MULTI-ORIENTATION EDGE-BASED SATELLITE IMAGE MATCHING METHOD FOR OPTICAL AND SAR IMAGES

Abstract. Multi-sensor image matching is a key technology to fully exploit complementary information on images from multiple sources. In particular, the nonlinear radiance distortion between optical and SAR images makes image matching very difficult. Focusing on satellite optical and SAR image matching, this paper proposed an oriented edge-based template image matching, namely Multiple Orientation Edge Feature Correlation (MOEFC). Firstly, the edge feature was extracted by the Sobel operator to construct oriented gradient channels on optical and SAR images. Then spatial filtering was performed separately by a two-dimensional and a one-dimensional kernel. Finally, the similarity was determined by stacking the normalized cross correlation (NCC) of each oriented gradient channel. To validate the new method, it was compared with other four edge-based template image matching methods detected by Sobel, Canny, Laplacian and phase congruency-based methods on satellite optical-SAR image pairs, which were tested in rural and urban areas. And the results show that the proposed MOEFC method can obtain the largest number of correct matching (NCM) with the least root mean square error (RMSE) in both areas. In the rural area, the NCM of the MOEFC method is higher than the other four methods, and the improvement ranges from 54.1% to 74.3%. And the RMSE of the MOEFC method is 0.645 pixel. In the urban area, the NCM of the MOEFC method can also be improved by 3.6% to 46.4%. And the RMSE of the MOEFC method equals 0.489 pixel.

Performance analysis of joint imaging system with polarized, infrared, and visible cameras for multi-sensor imaging

Polarized camera, infrared camera, and visible camera have their own advantages and disadvantages. Jointly imaging by these three cameras can effectively improve the detection and identification capabilities of the system. However, the performance of the joint imaging system differs from that of any single sub-camera when used in combination with several different cameras. To investigate the comprehensive performance of the imaging system when combined with several different cameras, isometric scaled plates from U.S. Navy resolution plate drawings were imaged at 5 m, 10 m, 15 m, and 20 m distances, and the results were analyzed using multiple quality evaluation metrics. Our work is instructive for the imaging applications of combined multi-source cameras.

SRIFD: A Shift Rotation Invariant Feature Descriptor for multi-sensor image matching

Feature descriptors such as SIFT (Scale Invariant Feature Transform) have been widely used in various computer vision applications, and many methods have been proposed to adapt the descriptors to multi-sensor images. Recently, RIFT (Radiation variation Insensitive Feature Transform) utilized the phase congruency that is invariant to the illumination and radiometric variation to construct descriptors for multi-sensor images. However, RIFT does not perform well on images of certain rotation degrees even with accurate main orientation of keypoints. To address this issue, this work designs a SoMIM (second-order maximum index map) by using the orientational response of Log-Gabor filters, and then constructs a novel descriptor, sRIFD (shift Rotation Invariant Featured Descriptor), without the computing main orientations. sRIFD performs evenly well on images of almost all rotation angles. To evaluate the performance of sRIFD, we use three image datasets, Infrared-Optical, Thermal-Optical, and Optical-Optical datasets. The quantitative results demonstrated that sRIFD performed better on multi-sensor image datasets under varying rotation angles.

Bounded PCA based Multi Sensor Image Fusion Employing Curvelet Transform Coefficients

The fusion of thermal and visible images acts as an important device for target detection. The quality of the spectral content of the fused image improves with wavelet-based image fusion. However, compared to PCA-based fusion, most wavelet-based methods provide results with a lower spatial resolution. The outcome gets better when the two approaches are combined, but they may still be refined. Compared to wavelets, the curvelet transforms more accurately depict the edges in the image. Enhancing the edges is a smart way to improve spatial resolution and the edges are crucial for interpreting the images. The fusion technique that utilizes curvelets enables the provision of additional data in both spectral and spatial areas concurrently. In this paper, we employ an amalgamation of Curvelet Transform and a Bounded PCA (CTBPCA) method to fuse thermal and visible images. To evidence the enhanced efficiency of our proposed technique, multiple evaluation metrics and comparisons with existing image merging methods are employed. Our approach outperforms others in both qualitative and quantitative analysis, except for runtime performance. Future Enhancement-The study will be based on using the fused image for target recognition. Future work should also focus on this method’s continued improvement and optimization for real-time video processing.

Robust multi-sensor image matching based on normalized self-similarity region descriptor

Multi-modal image matching is crucial in aerospace applications because it can fully exploit the complementary and valuable information contained in the amount and diversity of remote sensing images. However, it remains a challenging task due to significant non-linear radiometric, geometric differences, and noise across different sensors. To improve the performance of heterologous image matching, this paper proposes a normalized self-similarity region descriptor to extract consistent structural information. We first construct the pointwise self-similarity region descriptor based on the Euclidean distance between adjacent image blocks to reflect the structural properties of multi-modal images. Then, a linear normalization approach is used to form Modality Independent Region Descriptor (MIRD), which can effectively distinguish structural features such as points, lines, corners, and flat between multi-modal images. To further improve the matching accuracy, the included angle cosine similarity metric is adopted to exploit the directional vector information of multi-dimensional feature descriptors. The experimental results show that the proposed MIRD has better matching accuracy and robustness for various multi-modal image matching than the state-of-the-art methods. MIRD can effectively extract consistent geometric structure features and suppress the influence of SAR speckle noise using non-local neighboring image blocks operation, effectively applied to various multi-modal image matching.

Enhanced defect detection in NDE using registration aided heterogeneous data fusion

The degradation of systems in service, including pipelines, over time highlights the critical need for reliable and accurate defect detection to ensure safe operations. However, single modality-based Nondestructive Evaluation (NDE) data used in practical applications often suffers from noise contamination and errors caused by various factors like lift-off/standoff distances, probe drift, scanning speed, variation in data acquisition rates, and poor sensor sensitivity. The presence of agnostic noise types poses a challenge in extracting defect signals. To address these challenges, this paper presents an automated NDE theory-based data fusion framework aimed at enhancing the detection of surface and near-surface defects in magnetizable and conductive specimens. The Magnetic Flux Leakage (MFL) and Eddy Current (EC) based NDE sensing methods demonstrate the highly heterogeneous nature of noise distributions. Given the heterogeneity of the inspection methods, a screening rule is proposed to determine the conditions under which fusion would be beneficial. An important aspect of the proposed fusion method is registration, which ensures accurate alignment of multi-sensor image data. Two registration methods are proposed in this study as performing blind fusion without registration leads to erroneous results. The first registration method is translational, whereas the second method is registration based on linear optimal transport (OT) which proves to be effective in the boundary conditions. Finally, the registered source images from the EC and MFL modalities are fused using pixel-based fusion algorithms, including transform domain and spatial domain-based methods. Qualitative and quantitative assessments demonstrate that the registered fusion results exhibit higher accuracy and reliability compared to unregistered fused results and source images. Although the fusion method is applied to MFL and EC data in this paper, it is also suitable for other NDE modalities.

Editorial: Multi-sensor imaging and fusion: methods, evaluations, and applications

Editorial: Multi-sensor imaging and fusion: methods, evaluations, and applications