Fusion Module Research Articles

Multistage Spatial-Spectral Fusion Network for Spectral Super-Resolution.

Spectral super-resolution (SSR) aims to restore a hyperspectral image (HSI) from a single RGB image, in which deep learning has shown impressive performance. However, the majority of the existing deep-learning-based SSR methods inadequately address the modeling of spatial-spectral features in HSI. That is to say, they only sufficiently capture either the spatial correlations or the spectral self-similarity, which results in a loss of discriminative spatial-spectral features and hence limits the fidelity of the reconstructed HSI. To solve this issue, we propose a novel SSR network dubbed multistage spatial-spectral fusion network (MSFN). From the perspective of network design, we build a multistage Unet-like architecture that differentially captures the multiscale features of HSI both spatialwisely and spectralwisely. It consists of two types of the self-attention mechanism, which enables the proposed network to achieve global modeling of HSI comprehensively. From the perspective of feature alignment, we innovatively design the spatial fusion module (SpatialFM) and spectral fusion module (SpectralFM), aiming to preserve the comprehensively captured spatial correlations and spectral self-similarity. In this manner, the multiscale features can be better fused and the accuracy of reconstructed HSI can be significantly enhanced. Quantitative and qualitative experiments on the two largest SSR datasets (i.e., NTIRE2022 and NTIRE2020) demonstrate that our MSFN outperforms the state-of-the-art SSR methods. The code implementation will be uploaded at https://github.com/Matsuri247/MSFN-for-Spectral-Super-Resolution.

MSDA-HLGCformer-based context-aware fusion network for underwater organism detection

Underwater organism detection is crucial for marine ecology research and resource management. However, underwater images often suffer from issues like blurring, noise, and color distortion, making it challenging to distinguish targets from complex backgrounds. Existing detection methods struggle to separate targets in such conditions. To address these problems, we propose a Cross-Context Fusion Transformer YOLO network (CCFT-YOLO) for underwater organism detection. Specifically, we propose an MSDA-HLGCformer backbone network that captures and integrates local features at different depths and distances through the Multi-Scale Dilated Attention (MSDA) block at the low level, while at the high level, a hierarchical local–global context former (HLGCformer) is developed to contextualize the information at each scale, allowing the model to focus on the global context of the underwater environment. A context-aware cross-level fusion module (CACLFM) is designed to balance the retention of salient target and background texture information by integrating local and global information. Quantitative and qualitative experimental results show that CCFT-YOLO is competitive and robust in underwater organism detection compared to other state-of-the-art detectors.

Lightweight rail surface defect detection algorithm based on an improved YOLOv8

Existing deep learning methods for rail surface defect detection face issues such as poor compatibility with embedded detection systems, high computational resource consumption, and slow detection speeds. To address these challenges, we propose a lightweight rail surface defect detection algorithm11Code: https://github.com/haichao67/GD-YOLOv8. based on an improved YOLOv8.Inspired by Huawei’s Gold-YOLO, our algorithm redesigns the neck architecture, integrates feature extraction, fusion, and injection modules, and replaces Complete Intersection over Union with Scylla Intersection over Union. Comparative experiments confirm its efficacy, achieving 94.9% precision and 81.9% recall on the Rail Surface Defect Dataset, a 26.3% increase in frames per second, and a model size reduction to 63%. Our algorithm not only ensures satisfactory detection results but also reduces the number of model parameters and increases the detection speed.

TRRHA: A two-stream re-parameterized refocusing hybrid attention network for synthesized view quality enhancement

In multi-view video systems, the decoded texture video and its corresponding depth video are utilized to synthesize virtual views from different perspectives using the depth-image-based rendering (DIBR) technology in 3D-high efficiency video coding (3D-HEVC). However, the distortion of the compressed multi-view video and the disocclusion problem in DIBR can easily cause obvious holes and cracks in the synthesized views, degrading the visual quality of the synthesized views. To address this problem, a novel two-stream re-parameterized refocusing hybrid attention (TRRHA) network is proposed to significantly improve the quality of synthesized views. Firstly, a global multi-scale residual information stream is applied to extract the global context information by using refocusing attention module (RAM), and the RAM can detect the contextual feature and adaptively learn channel and spatial attention feature to selectively focus on different areas. Secondly, a local feature pyramid attention information stream is used to fully capture complex local texture details by using re-parameterized refocusing attention module (RRAM). The RRAM can effectively capture multi-scale texture details with different receptive fields, and adaptively adjust channel and spatial weights to adapt to information transformation at different sizes and levels. Finally, an efficient feature fusion module is proposed to effectively fuse the extracted global and local information streams. Extensive experimental results show that the proposed TRRHA achieves significantly better performance than the state-of-the-art methods. The source code will be available at https://github.com/647-bei/TRRHA.

CPDet: Circle-Permutation-Aware Object Detection for Heat Exchanger Cleaning

Shell–tube heat exchangers are commonly used equipment in large-scale industrial systems of wastewater heat exchange to reclaim the thermal energy generated during industrial processes. However, the internal surfaces of the heat exchanger tubes often accumulate fouling, which subsequently reduces their heat transfer efficiency. Therefore, regular cleaning is essential. We aim to detect circle holes on the end surface of the heat exchange tubes to further achieve automated positioning and cleaning tubes. Notably, these holes exhibit a regular distribution. To this end, we propose a circle-permutation-aware object detector for heat exchanger cleaning to sufficiently exploit prior information of the original inputs. Specifically, the interval prior to the extraction module extracts interval information among circle holes based on prior statistics, yielding prior interval context. The following interval prior fusion module slices original images into circle domain and background domain maps according to the prior interval context. For the circle domain map, prior-guided sparse attention using prior a circle–hole diameter as the step divides the circle domain map into patches and performs patch-wise self-attention. The background domain map is multiplied by a hyperparameter weak coefficient matrix. In this way, our method fully leverages prior information to selectively weigh the original inputs to achieve more effective hole detection. In addition, to adapt the hole shape, we adopt the circle representation instead of the rectangle one. Extensive experiments demonstrate that our method achieves state-of-the-art performance and significantly boosts the YOLOv8 baseline by 5.24% mAP50 and 5.25% mAP50:95.

Classification and extraction method of hidden dangers along railway lines based on semantic segmentation network

ABSTRACT Foreign objects invading high-speed railway lines can cause danger. One existing solution is to use remote sensing images to analyse the dangerous areas along the railway line, thereby providing a certain amount of investigation time. Considering the spatial and temporal resolution characteristics of existing remote sensing technologies in identifying floating objects and the reality of rapid land use changes, this paper identifies areas on the ground where floating objects may be generated by using semantic segmentation techniques oriented to remotely sensed imagery and provides early warnings to staff along the route. However, these regions that need to be analysed have different semantics and scales. To address these challenges, this paper proposes a Dual-branch Parallel Fusion Network (DPFNet) based on Transformer, aimed at enhancing multi-class semantic segmentation in remote sensing images. To leverage global contextual information, we introduce a Swin Transformer-based backbone network, which employs self-attention to capture a comprehensive scene context, facilitating better segmentation by considering the entire scene’s context. For multi-scale semantic features, we propose one approach that involves independent branching feature expression and a Multi-scale Feature Space Fusion Module (MFSFM). The former enriches multi-scale information, while the latter fuses features across different levels to capture diverse semantic features. Experimental results demonstrate that DPFNet can effectively identify the hidden danger area, and the fusion of multi-scale features makes the network more accurately identify and segment the risk area of different sizes, improving the segmentation accuracy and robustness, and is of great significance to the formation of the ‘prevention’ as the core of the railway safety operation.

Spatial–Temporal-Correlation-Constrained Dynamic Graph Convolutional Network for Traffic Flow Forecasting

Accurate traffic flow prediction in road networks is essential for intelligent transportation systems (ITS). Since traffic data are collected from the road network with spatial topological and time series sequences, the traffic flow prediction is regarded as a spatial–temporal prediction task. With the powerful ability to model the non-Euclidean data, the graph convolutional network (GCN)-based models have become the mainstream framework for traffic forecasting. However, existing GCN-based models either use the manually predefined graph structure to capture the spatial features, ignoring the heterogeneity of road networks, or simply perform 1-D convolution with fixed kernel to capture the temporal dependencies of traffic data, resulting in insufficient long-term temporal feature extraction. To solve those issues, a spatial–temporal correlation constrained dynamic graph convolutional network (STC-DGCN) is proposed for traffic flow forecasting. In STC-DGCN, a spatial–temporal embedding encoder module (STEM) is first constructed to encode the dynamic spatial relationships for road networks at different time steps. Then, a temporal feature encoder module with heterogeneous time series correlation modeling (TFE-HCM) and a spatial feature encoder module with dynamic multi-graph modeling (SFE-DCM) are designed to generate dynamic graph structures for effectively capturing the dynamic spatial and temporal correlations. Finally, a spatial–temporal feature fusion module based on a gating fusion mechanism (STM-GM) is proposed to effectively learn and leverage the inherent spatial–temporal relationships for traffic flow forecasting. Experimental results from three real-world traffic flow datasets demonstrate the superior performance of the proposed STC-DGCN compared with state-of-the-art traffic flow forecasting models.

TransCDR: a deep learning model for enhancing the generalizability of drug activity prediction through transfer learning and multimodal data fusion

BackgroundAccurate and robust drug response prediction is of utmost importance in precision medicine. Although many models have been developed to utilize the representations of drugs and cancer cell lines for predicting cancer drug responses (CDR), their performances can be improved by addressing issues such as insufficient data modality, suboptimal fusion algorithms, and poor generalizability for novel drugs or cell lines.ResultsWe introduce TransCDR, which uses transfer learning to learn drug representations and fuses multi-modality features of drugs and cell lines by a self-attention mechanism, to predict the IC50 values or sensitive states of drugs on cell lines. We are the first to systematically evaluate the generalization of the CDR prediction model to novel (i.e., never-before-seen) compound scaffolds and cell line clusters. TransCDR shows better generalizability than 8 state-of-the-art models. TransCDR outperforms its 5 variants that train drug encoders (i.e., RNN and AttentiveFP) from scratch under various scenarios. The most critical contributors among multiple drug notations and omics profiles are Extended Connectivity Fingerprint and genetic mutation. Additionally, the attention-based fusion module further enhances the predictive performance of TransCDR. TransCDR, trained on the GDSC dataset, demonstrates strong predictive performance on the external testing set CCLE. It is also utilized to predict missing CDRs on GDSC. Moreover, we investigate the biological mechanisms underlying drug response by classifying 7675 patients from TCGA into drug-sensitive or drug-resistant groups, followed by a Gene Set Enrichment Analysis.ConclusionsTransCDR emerges as a potent tool with significant potential in drug response prediction.

PRN: progressive reasoning network and its image completion applications

Ancient murals embody profound historical, cultural, scientific, and artistic values, yet many are afflicted with challenges such as pigment shedding or missing parts. While deep learning-based completion techniques have yielded remarkable results in restoring natural images, their application to damaged murals has been unsatisfactory due to data shifts and limited modeling efficacy. This paper proposes a novel progressive reasoning network designed specifically for mural image completion, inspired by the mural painting process. The proposed network comprises three key modules: a luminance reasoning module, a sketch reasoning module, and a color fusion module. The first two modules are based on the double-codec framework, designed to infer missing areas’ luminance and sketch information. The final module then utilizes a paired-associate learning approach to reconstruct the color image. This network utilizes two parallel, complementary pathways to estimate the luminance and sketch maps of a damaged mural. Subsequently, these two maps are combined to synthesize a complete color image. Experimental results indicate that the proposed network excels in restoring clearer structures and more vivid colors, surpassing current state-of-the-art methods in both quantitative and qualitative assessments for repairing damaged images. Our code and results will be publicly accessible at https://github.com/albestobe/PRN.

LapUNet: a novel approach to monocular depth estimation using dynamic laplacian residual U-shape networks

Monocular depth estimation is an important but challenging task. Although the performance has been improved by adopting various encoder-decoder architectures, the estimated depth maps lack structure details and clear edges due to simple repeated upsampling. To solve this problem, this paper presents the novel LapUNet (Laplacian U-shape networks), in which the encoder adopts ResNeXt101, and the decoder is constructed with the novel DLRU (dynamic Laplacian residual U-shape) module. The DLRU module based on the U-shape structure can supplement high-frequency features by fusing dynamic Laplacian residual into the process of upsampling, and the residual is dynamically learnable due to the addition of convolutional operation. Also, the ASPP (atrous spatial pyramid pooling) module is introduced to capture image context at multiple scales though multiple parallel atrous convolutional layers, and the depth map fusion module is used for combining high and low frequency features from depth maps with different spatial resolution. Experiments demonstrate that the proposed model with moderate model size is superior to other previous competitors on the KITTI and NYU Depth V2 datasets. Furthermore, 3D reconstruction and target ranging by utilizing the estimated depth maps prove the effectiveness of our proposed method.

Dense projection fusion for 3D object detection

Fusing information from LiDAR and cameras can effectively enhance the overall perceptivity of autonomous vehicles in various scenarios. Despite the relatively good results achieved by point-wise fusion and Bird’s-Eye-View (BEV) fusion, they still cannot fully leverage the image information and lack of effective depth information. For any fusion methods, the multi-modal features first need to be concatenated along the channel, and then the fused features are extracted using convolutional layers. However, this type of fusion methods is effective, but too coarse which causes that the fused features cannot pay more attention to the regions with important features and suffer from severe noise. To tackle these issues, we propose in this paper a Dense Projection Fusion (DPFusion) approach. It consists of two new modules: dense depth map guided BEV transform (DGBT) module and multi-modal feature adaptive fusion (MFAF) module. The DGBT module first quickly estimates the depth of each pixel and then projects all image features to the BEV space, making full use of the image information. The MFAF module computes the image weights and point cloud weights for each channel in each BEV grid and then adaptively weights and fuses the image BEV features with the point cloud BEV features. It is worth pointing out that the MFAF module makes the fused features pay more attention to background outline and object outline. Our proposed DPFusion demonstrates competitive results in 3D object detection, achieving a mean Average Precision (mAP) of 70.4 and a nuScenes detection score (NDS) of 72.3 on the nuScenes validation set.

YOLOrot2.0: A novel algorithm for high-precision rice seed size measurement with real-time processing

We present YOLOrot2.0, an automated algorithm for measuring the dimensions of rice seeds, which holds significant importance in evaluating grain quality and genetic breeding research. The current methods for rice seed measurement exhibit various limitations, such as the need for seed preprocessing and selection, specialized equipment for measurement, and prolonged measurement time. To address these issues, YOLOrot2.0 introduces the following improvements: Firstly, it utilizes an anchor-free detection algorithm to optimize the detection of rotated targets. Secondly, the utilization of the Kalman Filter IoU loss function, which combines the horizontal bounding box and smooth L1 loss function, accelerates the convergence speed of the network. Moreover, the YOLOv8 architecture was adjusted by modifying certain convolutional layers and the Context to Fusion module to enhance the detection capabilities for smaller targets. Finally, YOLOrot2.0 enables direct processing of the entire image, eliminating the need for image partitioning and resulting in substantial time and processing savings. Experimental results demonstrate that YOLOrot2.0 achieves an mAP (mean average precision) score of 0.95, surpassing other comparative algorithms. Additionally, YOLOrot2.0 provides highly accurate measurements of seed length and width, closely aligned with ground truth values. Notably, YOLOrot2.0 achieves real-time detection by achieving an average detection time of only 11.5 ms for images sized 3000×4000 pixels.

OGRN: optical-guided residual dense network for SAR image super-resolution reconstruction network

ABSTRACT Although Convolutional Neural Networks have significantly improved the development of SAR image super-resolution (SR) technology in recent years, it is a very challenging problem to reconstruct SAR image with large-scale factors, such as ×4 and ×8 due to limited available information from low-resolution image. The co-registered high-resolution optical image has been successfully applied to enhance the quality of SAR image due to its discriminative characteristics. Compared with single-frame SAR image SR reconstruction technology, optical image-guided SAR image SR reconstruction technology has better performance. This paper proposes an optical-guided residual dense network (OGRN) for SAR image super-resolution reconstruction network. The network is an end-to-end network. Its most important characteristic is that the SR model of SAR images can be obtained through optical image assistance during training, and high-quality SAR images can be generated without optical image assistance during testing. Firstly, a new dense residual backbone network that extracts high-frequency information is constructed. It can extract and propagate high-frequency features of SAR images efficiently to improve the feature representation ability of the network for high-frequency features. Then, the extracted high-frequency features of the SAR image are used to generate an optical image through the designed SAR-to-Optical image translation network. Finally, a reconstruction module is constructed based on the proposed fusion module for optical and SAR images. The fusion module fuses the high-frequency feature information of the extracted optical images with the SAR image features to provide features for SAR image SR reconstruction. Extensive experiments conducted on Sen1-2 and QXS datasets demonstrated that under the guidance of optical image, our OGRN can achieve excellent performance in both quantitative assessment metrics and visual quality.

Contraband detection algorithm for X‐ray security inspection images based on global semantic enhancement

AbstractAiming at the problem of low detection accuracy caused by overlapping occlusion and noise disturbance, a contraband detection algorithm for X‐ray security inspection images based on global semantic enhancement is proposed to achieve accurate contraband target detection by enhancing global semantic information. First, the disturbance suppression module is used to weaken the noise disturbance at different positions by local suppression and aggregate finer detail information. Then, the parallel cascade search module is used to capture long‐range dependencies and strengthen the representation of global semantic information, which helps the model identify contraband under overlapping occlusion. Finally, the contribution of different features is adaptively adjusted through the feature‐weighted fusion module, which promotes the effective fusion of multi‐scale features and improves the accuracy of model detection. The method in this article has been extensively evaluated and experimented on three mainstream benchmark datasets: SIXray, OPIXray, and PIDray. The mAPs of three datasets reach 93.5%, 91.9%, and 85.9%, respectively. The experimental results fully demonstrate that the method in this article has better performance compared with the latest method, which can meet the practical application requirements of real‐time target detection.

Use of SNOMED CT in Large Language Models: Scoping Review.

Large language models (LLMs) have substantially advanced natural language processing (NLP) capabilities but often struggle with knowledge-driven tasks in specialized domains such as biomedicine. Integrating biomedical knowledge sources such as SNOMED CT into LLMs may enhance their performance on biomedical tasks. However, the methodologies and effectiveness of incorporating SNOMED CT into LLMs have not been systematically reviewed. This scoping review aims to examine how SNOMED CT is integrated into LLMs, focusing on (1) the types and components of LLMs being integrated with SNOMED CT, (2) which contents of SNOMED CT are being integrated, and (3) whether this integration improves LLM performance on NLP tasks. Following the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines, we searched ACM Digital Library, ACL Anthology, IEEE Xplore, PubMed, and Embase for relevant studies published from 2018 to 2023. Studies were included if they incorporated SNOMED CT into LLM pipelines for natural language understanding or generation tasks. Data on LLM types, SNOMED CT integration methods, end tasks, and performance metrics were extracted and synthesized. The review included 37 studies. Bidirectional Encoder Representations from Transformers and its biomedical variants were the most commonly used LLMs. Three main approaches for integrating SNOMED CT were identified: (1) incorporating SNOMED CT into LLM inputs (28/37, 76%), primarily using concept descriptions to expand training corpora; (2) integrating SNOMED CT into additional fusion modules (5/37, 14%); and (3) using SNOMED CT as an external knowledge retriever during inference (5/37, 14%). The most frequent end task was medical concept normalization (15/37, 41%), followed by entity extraction or typing and classification. While most studies (17/19, 89%) reported performance improvements after SNOMED CT integration, only a small fraction (19/37, 51%) provided direct comparisons. The reported gains varied widely across different metrics and tasks, ranging from 0.87% to 131.66%. However, some studies showed either no improvement or a decline in certain performance metrics. This review demonstrates diverse approaches for integrating SNOMED CT into LLMs, with a focus on using concept descriptions to enhance biomedical language understanding and generation. While the results suggest potential benefits of SNOMED CT integration, the lack of standardized evaluation methods and comprehensive performance reporting hinders definitive conclusions about its effectiveness. Future research should prioritize consistent reporting of performance comparisons and explore more sophisticated methods for incorporating SNOMED CT's relational structure into LLMs. In addition, the biomedical NLP community should develop standardized evaluation frameworks to better assess the impact of ontology integration on LLM performance.

SOD‐YOLO: Small Object Detection Network for UAV Aerial Images

With the rapid development of the UAV industry, object detection using UAV has become a research hotspot. However, most current object detection models based on deep learning have large parameter counts and are difficult to deploy on embedded devices with limited memory and computational power. To address this problem, a Small Object Detection network for UAV aerial images SOD‐YOLO based on YOLOv8 is proposed, which can meet the application requirements of resource‐constrained devices while ensuring the detection accuracy. First, cross‐domain fusion attention (CDFA) mechanism is proposed to build the C2f‐Attention module in this paper, which is embedded in the backbone network in order to improve the extraction capability of key object features. Meanwhile, the AIFI_LSPE feature fusion module with improved RT‐DETR and the IoU‐aware query selection mechanism are added to the path aggregation network to improve the accuracy of multi‐scale object detection. In addition, in order to balance the sample size ratio and improve the robustness of the network model, we make a new UAV image dataset named VisDrone2019 Extended Edition (VDEE) using images from the VisDrone2019 and UAVDT public datasets. Finally, Shape‐IoU is used as a loss function to reduce the difference between the object GT frame and the detection frame. Experiments show that SOD‐YOLO has a mAP@0.5 of 42.8% in the VDEE dataset, which is increased by 5.1% over YOLOv8. In the VisDrone2019 dataset mAP@0.5 is 39.2%, an improvement of 5.8% over YOLOv8. © 2024 Institute of Electrical Engineers of Japan and Wiley Periodicals LLC.

Feature Pyramid Network Based Spatial Attention and Cross‐Level Semantic Similarity for Diseases Segmentation From Capsule Endoscopy Images

ABSTRACTAs an emerging technology that uses a pill‐sized camera to visualize images of the digestive tract. Wireless capsule endoscopy (WCE) presents several advantages, since it is far less invasive, does not need sedation and has less possible complications compared to standard endoscopy. Hence, it might be exploited as alternative to the standard procedure. WCE is used to diagnosis a variety of gastro‐intestinal diseases such as polyps, ulcers, crohns disease, and hemorrhages. Nevertheless, WCE videos produced after a test may consist of thousands of frames per patient that must be viewed by medical specialists, besides, the capsule free mobility and technological limits cause production of a low quality images. Hence, development of an automatic tool based on artificial intelligence might be very helpful. Moreover, most state‐of‐the‐art works aim at images classification (normal/abnormal) while ignoring diseases segmentation. Therefore, in this work a novel method based on Feature Pyramid Network model is presented. This approach aims at diseases segmentation from WCE images. In this model, modules to optimize and combine features were employed. Specifically, semantic and spatial features were mutually compensated by spatial attention and cross‐level global feature fusion modules. The proposed method testing F1‐score and mean intersection over union are 94.149% and 89.414%, respectively, in the MICCAI 2017 dataset. In the KID Atlas dataset, the method achieved a testing F1‐score and mean intersection over union of 94.557% and 90.416%, respectively. Through the performance analysis, the mean intersection over union in the MICCAI 2017 dataset is 20.414%, 18.484%, 11.444%, 8.794% more than existing approaches. Moreover, the proposed scheme surpassed the methods used for comparison by 29.986% and 9.416% in terms of mean intersection over union in KID Atlas dataset. These results indicate that the proposed approach is promising in diseases segmentation from WCE images.

ACP-Net: Asymmetric Center Positioning Network for Real-Time Text Detection

Scene text detection is crucial across numerous application fields. However, despite the emphasis on real-time performance in scene text detection, most existing detection models utilize the Feature Pyramid Network (FPN) for feature extraction, often disregarding its inherent limitations. Integrating high-resolution multi-channel features into FPN requires substantial computational resources. While FPN treats local and global features equally and is stable in various applications, its suitability for text-specific features is questionable. To this end, we propose the Asymmetric Center Positioning Network (ACP-Net) to replace FPN, achieving accuracy and real-time text detection in complex scenarios. ACP-Net features an asymmetric feature structure with independent branches for global and local information, along with an adaptive weighted fusion module to capture long-range dependencies effectively. In addition, a text center positioning module enhances text feature understanding by learning feature centers. Comprehensive evaluations across various terminals confirmed ACP-Net’s superior accuracy and speed.

E2TNet: Efficient enhancement Transformer network for hyperspectral image classification

Recently, Convolutional Transformer-based models have become popular in hyperspectral image (HSI) classification tasks and gained competitive classification performance. However, some Convolutional Transformer-based models fail to effectively mine the global correlations of coarse-grained and fine-grained features, which is adverse to recognizing the refined scale variation information of land-cover. The combination of convolution operations and multihead self-attention mechanisms also increases the computational cost, leading to low classification efficiency. In addition, shallow spectral–spatial features are directly input into the encoder, which inevitably incurs redundant spectral information. Therefore, this paper proposes an efficient enhancement Transformer network (E2TNet) for HSI classification. Specifically, this paper first designs a spectral–spatial feature fusion module to extract spectral and spatial features from HSI cubes and fuse them. Second, considering that redundant spectral information has a negative impact on classification performance, this paper designs a spectral–spatial feature weighted module to improve the feature representation of critical spectral information. Finally, to explore the global correlations of coarse-grained and fine-grained features and improve classification efficiency, an efficient multigranularity information fusion module is embedded in the encoder of E2TNet. The experiment is conducted on four benchmark hyperspectral datasets, and the experimental results demonstrate that the proposed E2TNet is better than several Convolutional Transformer-based classification models in terms of classification accuracy and classification efficiency.

SFINet: A semantic feature interactive learning network for full-time infrared and visible image fusion

Infrared and visible image fusion aims to combine data from various source images to generate a high-quality image. Nevertheless, numerous fusion methods often prioritize visual quality above semantic information. To address this problem, we present a Semantic Feature Interactive Learning Network (SFINet) for full-time infrared and visible images. The SFINet encompasses an image fusion network and an image segmentation network through a Semantic Feature Interaction (SFI) module. The image fusion network employs Multi-scale Feature Extraction (MFE) modules to capture global and local information at multiple scales. Meanwhile, it performs an adaptive fusion of complementary information using a Dual Attention Feature Fusion (DAFF) module. The image segmentation network guides the image fusion network using the SFI module for semantic feature interaction. Comparative results prove that the proposed method is superior to state-of-the-art (SOTA) models in image fusion and semantic segmentation tasks. The code is available at https://github.com/songwenhao123/SFINet.