Encoder-decoder Architecture Research Articles

A Deformable and Multi-Scale Network with Self-Attentive Feature Fusion for SAR Ship Classification

The identification of ships in Synthetic Aperture Radar (SAR) imagery is critical for effective maritime surveillance. The advent of deep learning has significantly improved the accuracy of SAR ship classification and recognition. However, distinguishing features between different ship categories in SAR images remains a challenge, particularly as the number of categories increases. The key to achieving high recognition accuracy lies in effectively extracting and utilizing discriminative features. To address this, we propose DCN-MSFF-TR, a novel recognition model inspired by the Transformer encoder–decoder architecture. Our approach integrates a deformable convolutional module (DCN) within the backbone network to enhance feature extraction. Additionally, we introduce multi-scale self-attention processing from the Transformer into the feature hierarchy and fuse these representations at appropriate levels using a feature pyramid strategy. This enables each layer to leverage both its own information and synthesized features from other layers, enhancing feature representation. Extensive evaluations on the OpenSARShip-3-Complex and OpenSARShip-6-Complex datasets demonstrate the effectiveness of our method. DCN-MSFF-TR achieves average recognition accuracies of 78.1% and 66.7% on the three-class and six-class datasets, respectively, outperforming existing recognition models and showcasing its superior capability in accurately identifying ship categories in SAR images.

FluPMT: Prediction of Predominant Strains of Influenza A Viruses via Multi-Task Learning.

Seasonal influenza vaccines play a crucial role in saving numerous lives annually. However, the constant evolution of the influenza A virus necessitates frequent vaccine updates to ensure its ongoing effectiveness. The decision to develop a new vaccine strain is generally based on the assessment of the current predominant strains. Nevertheless, the process of vaccine production and distribution is very time-consuming, leaving a window for the emergence of new variants that could decrease vaccine effectiveness, so predictions of influenza A virus evolution can inform vaccine evaluation and selection. Hence, we present FluPMT, a novel sequence prediction model that applies an encoder-decoder architecture to predict the hemagglutinin (HA) protein sequence of the upcoming season's predominant strain by capturing the patterns of evolution of influenza A viruses. Specifically, we employ time series to model the evolution of influenza A viruses, and utilize attention mechanisms to explore dependencies among residues of sequences. Additionally, antigenic distance prediction based on graph network representation learning is incorporated into the sequence prediction as an auxiliary task through a multi-task learning framework. Experimental results on two influenza datasets highlight the exceptional predictive performance of FluPMT, offering valuable insights into virus evolutionary dynamics, as well as vaccine evaluation and production.

Object Counting via Group and Graph Attention Network.

Object counting, defined as the task of accurately predicting the number of objects in static images or videos, has recently attracted considerable interest. However, the unavoidable presence of background noise prevents counting performance from advancing further. To address this issue, we created a group and graph attention network (GGANet) for dense object counting. GGANet is an encoder-decoder architecture incorporating a group channel attention (GCA) module and a learnable graph attention (LGA) module. The GCA module groups the feature map into several subfeatures, each of which is assigned an attention factor through the identical channel attention. The LGA module views the feature map as a graph structure in which the different channels represent diverse feature vertices, and the responses between channels represent edges. The GCA and LGA modules jointly avoid the interference of irrelevant pixels and suppress the background noise. Experiments are conducted on four crowd-counting datasets, two vehicle-counting datasets, one remote-sensing counting dataset, and one few-shot object-counting dataset. Comparative results prove that the proposed GGANet achieves superior counting performance.

HDCCT: Hybrid Densely Connected CNN and Transformer for Infrared and Visible Image Fusion

Multi-modal image fusion is a methodology that combines image features from multiple types of sensors, effectively improving the quality and content of fused images. However, most existing deep learning fusion methods need to integrate global or local features, restricting the representation of feature information. To address this issue, a hybrid densely connected CNN and transformer (HDCCT) fusion framework is proposed. In the proposed HDCCT framework, the network of the CNN-based blocks obtain the local structure of the input data, and the transformer-based blocks obtain the global structure of the original data, significantly improving the feature representation. In the fused image, the proposed encoder–decoder architecture is designed for both the CNN and transformer blocks to reduce feature loss while preserving the characterization of all-level features. In addition, the cross-coupled framework facilitates the flow of feature structures, retains the uniqueness of information, and makes the transform model long-range dependencies based on the local features already extracted by the CNN. Meanwhile, to retain the information in the source images, the hybrid structural similarity (SSIM) and mean square error (MSE) loss functions are introduced. The qualitative and quantitative comparisons of grayscale images with infrared and visible image fusion indicate that the suggested method outperforms related works.

Hierarchical stacked spatiotemporal self-attention network for sea surface temperature forecasting

Sea surface temperature (SST) is a highly complex spatiotemporal variable, which stems from its susceptibility to non-linear dynamical processes and substantial spatiotemporal variability. In particular, accurately forecasting small-scale SST is a formidable challenge due to the compounded effects of diverse physical processes spanning across various scales. In this study, we employ deep learning methods to mine the ocean’s evolutionary patterns, as the ocean’s dynamic mechanisms are inherently embedded in spatiotemporal data. We propose a hierarchical stacked spatiotemporal self-attention mechanism (HSSSA) network architecture. The hierarchical stacked encoder–decoder architecture provides the capability for feature fusion and extraction at different scales. The spatial self-attention and temporal self-attention modules simultaneously focus on information from different spatial locations and time steps, allowing the exploration of spatiotemporal patterns in the complex dynamics of the ocean. The experiments are conducted on a high-resolution East China Sea dataset (1/10°×1/10°) to demonstrate the forecast performance of the proposed model for refined ocean variables. The 15-day forecasts indicate that the HSSSA method outperforms the EOF-ARIMA and CNN-Transformer methods.

URINet: Unsupervised point cloud rotation invariant representation learning via semantic and structural reasoning

In recent years, many rotation-invariant networks have been proposed to alleviate the interference caused by point cloud arbitrary rotations. These networks have demonstrated powerful representation learning capabilities. However, most of those methods rely on costly manually annotated supervision for model training. Moreover, they fail to reason the structural relations and lose global information. To address these issues, we present an unsupervised method for achieving comprehensive rotation invariant representations without human annotation. Specifically, we propose a novel encoder–decoder architecture named URINet, which learns a point cloud representation by combining local semantic and global structural information, and then reconstructs the input without rotation perturbation. In detail, the encoder is a two-branch network where the graph convolution based structural branch models the relationships among local regions to learn global structural knowledge and the semantic branch learns rotation invariant local semantic features. The two branches derive complementary information and explore the point clouds comprehensively. Furthermore, to avoid the self-reconstruction ambiguity brought by uncertain poses, a bidirectional alignment is proposed to measure the quality of reconstruction results without orientation knowledge. Extensive experiments on downstream tasks show that the proposed method significantly surpasses existing state-of-the-art methods on both synthetic and real-world datasets.

Relation extraction for colorectal cancer via deep learning with entity-aware feature orthogonal decomposition

Relation extraction is significant for text structuring of colorectal cancer (CRC) pathological reports to facilitate doctors’ disease diagnoses. Although many relation extraction methods have been extensively studied for various natural language processing applications, they cannot be well transferred to be applied for CRC pathological reports since CRC pathological reports have some unique characteristics. To this end, a deep learning framework is designed in this paper to extract entity relations in CRC pathological reports, which is based on an encoder–decoder architecture with entity-aware feature orthogonal decomposition. Specifically, to effectively extract semantic features of long and short entities, a two-stream encoder is designed based on an edge-aware convolutional neural network and a dimension-aware dilated convolution residual network. To alleviate the influence of the blending of subject–object features, entity-aware feature orthogonal decomposition is designed to decompose the extracted semantic features into three types, i.e. subject features, object features and subject–object shared features. A stage-wise cross entropy loss is proposed to well train the network. Comparison experiments indicated that our designed network performs well on CRC pathological texts with the performance of 92.3% F1 score, 93.1% Precision, and 91.5% Recall, outperforming the existing relation extraction models.

Domain knowledge-driven encoder–decoder for nasopharyngeal carcinoma segmentation

Accurate image segmentation is vital for the diagnosis and treatment of nasopharyngeal carcinoma (NPC). Although deep neural networks have shown promising performance in NPC segmentation, they depend on large-scale pixel-level annotation datasets for purely data-driven model training, which poses practical limitations. To address this challenge, this paper explores the domain knowledge, namely the expert knowledge from radiologists, and proposes a domain knowledge-driven encoder–decoder architecture. Specifically, the domain knowledge on image spatial information is formulated as Gaussian mixture distribution and then transformed into an optimal transport-based expert-prior regularization of the encoder, which enhances the model’s ability in capturing discriminative features. To project the encoded features onto pixel space and obtain the segmentation maps, a cross-scale feature refinement module is built in the decoder with theoretical justification, which integrates the domain knowledge that radiologists segment NPC in a gradual refinement process. Experimental results verify the effectiveness of the proposed method for NPC segmentation. Remarkably, despite using only 10% of pixel-level annotation data, the domain knowledge-driven model outperforms recent deep neural networks that use the entire training data.

RDMS: Reverse distillation with multiple students of different scales for anomaly detection

AbstractUnsupervised anomaly detection, often approached as a one‐class classification problem, is a critical task in computer vision. Knowledge distillation has emerged as a promising technique for enhancing anomaly detection accuracy, especially with the advent of reverse distillation networks that employ encoder–decoder architectures. This study introduces a novel reverse knowledge distillation framework known as RDMS, which incorporates a pretrained teacher encoding module, a multi‐level feature fusion connection module, and a student decoding module consisting of three independent decoders. RDMS is designed to distill distinct features from the teacher encoder, mitigating overfitting issues associated with similar or identical teacher–student structures. The model achieves an average of 99.3 image‐level AUROC and 98.34 pixel‐level AUROC on the MVTec‐AD dataset and demonstrates state‐of‐the‐art performance on the more challenging BTAD dataset. The RDMS model's high accuracy in anomaly detection and localization underscores the potential of multi‐student reverse distillation to advance unsupervised anomaly detection capabilities. The source code is available at https://github.com/zihengchen777/RDMS

MSFE-UIENet: A Multi-Scale Feature Extraction Network for Marine Underwater Image Enhancement

Underwater optical images have outstanding advantages for short-range underwater target detection tasks. However, owing to the limitations of special underwater imaging environments, underwater images often have several problems, such as noise interference, blur texture, low contrast, and color distortion. Marine underwater image enhancement addresses degraded underwater image quality caused by light absorption and scattering. This study introduces MSFE-UIENet, a high-performance network designed to improve image feature extraction, resulting in deep-learning-based underwater image enhancement, addressing the limitations of single convolution and upsampling/downsampling techniques. This network is designed to enhance the image quality in underwater settings by employing an encoder–decoder architecture. In response to the underwhelming enhancement performance caused by the conventional networks’ sole downsampling method, this study introduces a pyramid downsampling module that captures more intricate image features through multi-scale downsampling. Additionally, to augment the feature extraction capabilities of the network, an advanced feature extraction module was proposed to capture detailed information from underwater images. Furthermore, to optimize the network’s gradient flow, forward and backward branches were introduced to accelerate its convergence rate and improve stability. Experimental validation using underwater image datasets indicated that the proposed network effectively enhances underwater image quality, effectively preserving image details and noise suppression across various underwater environments.

DAEiS-Net: Deep Aggregation Network with Edge Information Supplement for Tunnel Water Stain Segmentation.

Tunnel disease detection and maintenance are critical tasks in urban engineering, and are essential for the safety and stability of urban transportation systems. Water stain detection presents unique challenges due to its variable morphology and scale, which leads to insufficient multiscale contextual information extraction and boundary information loss in complex environments. To address these challenges, this paper proposes a method called Deep Aggregation Network with Edge Information Supplement (DAEiS-Net) for detecting tunnel water stains. The proposed method employs a classic encoder-decoder architecture. Specifically, in the encoder part, a Deep Aggregation Module (DAM) is introduced to enhance feature representation capabilities. Additionally, a Multiscale Cross-Attention Module (MCAM) is proposed to suppress noise in the shallow features and enhance the texture information of the high-level features. Moreover, an Edge Information Supplement Module (EISM) is designed to mitigate semantic gaps across different stages of feature extraction, improving the extraction of water stain edge information. Furthermore, a Sub-Pixel Module (SPM) is proposed to fuse features at various scales, enhancing edge feature representation. Finally, we introduce the Tunnel Water Stain Dataset (TWS), specifically designed for tunnel water stain segmentation. Experimental results on the TWS dataset demonstrate that DAEiS-Net achieves state-of-the-art performance in tunnel water stain segmentation.

What Kind of Transformer Models to Use for the ICD-10 Codes Classification Task.

Coding according to the International Classification of Diseases (ICD)-10 and its clinical modifications (CM) is inherently complex and expensive. Natural Language Processing (NLP) assists by simplifying the analysis of unstructured data from electronic health records, thereby facilitating diagnosis coding. This study investigates the suitability of transformer models for ICD-10 classification, considering both encoder and encoder-decoder architectures. The analysis is performed on clinical discharge summaries from the Medical Information Mart for Intensive Care (MIMIC)-IV dataset, which contains an extensive collection of electronic health records. Pre-trained models such as BioBERT, ClinicalBERT, ClinicalLongformer, and ClinicalBigBird are adapted for the coding task, incorporating specific preprocessing techniques to enhance performance. The findings indicate that increasing context length improves accuracy, and that the difference in accuracy between encoder and encoder-decoder models is negligible.

Enhanced deep learning network for accurate digital elevation model generation from LiDAR data

This paper presents a unique hierarchical deep network to tackle the task of identifying and filtering non-ground objects from point cloud data. This task is essential in the building of digital terrain models (DTMs). The proposed network is based on a deep encoder-decoder architecture and includes efficient convolutional connections to improve the identification of items that are not on the ground. In this architectural framework, a block for extracting features is intentionally created to capture a wide range of characteristics at many levels. Additionally, a technique for fusing global and local data is included to further enhance the accuracy of detection. The effectiveness of the proposed deep network in accurately detecting objects is validated by a comparative study with current approaches, utilizing ISPRS data. This analysis demonstrates the superiority of the proposed network in terms of object detection accuracy.

Multi-modal transformer architecture for medical image analysis and automated report generation

Medical practitioners examine medical images, such as X-rays, write reports based on the findings, and provide conclusive statements. Manual interpretation of the results and report generation by examiners are time-consuming processes that lead to potential delays in diagnosis. We propose an automated report generation model for medical images leveraging an encoder–decoder architecture. Our model utilizes transformer architectures, including Vision Transformer (ViT) and its variants like Data Efficient Image Transformer (DEiT) and BERT pre-training image transformer (BEiT), as an encoder. These transformers are adapted for processing to extract and gain visual information from medical images. Reports are transformed into text embeddings, and the Generative Pre-trained Transformer (GPT2) model is used as a decoder to generate medical reports. Our model utilizes a cross-attention mechanism between the vision transformer and GPT2, which enables it to create detailed and coherent medical reports based on the visual information extracted by the encoder. In our model, we have extended the report generation with general knowledge, which is independent of the inputs and provides a comprehensive report in a broad sense. We conduct our experiments on the Indiana University X-ray dataset to demonstrate the effectiveness of our models. Generated medical reports from the model are evaluated using word overlap metrics such as Bleu scores, Rouge-L, retrieval augmentation answer correctness, and similarity metrics such as skip thought cs, greedy matching, vector extrema, and RAG answer similarity. Results show that our model is performing better than the recurrent models in terms of report generation, answer similarity, and word overlap metrics. By automating the report generation process and incorporating advanced transformer architectures and general knowledge, our approach has the potential to significantly improve the efficiency and accuracy of medical image analysis and report generation.

A traffic-weather generative adversarial network for traffic flow prediction for road networks under bad weather

Traffic flow prediction is pivotal in providing reliable information for intelligent traffic systems. Unexpected events, such as bad weather, unavoidably impact the precision of traffic flow prediction. Therefore, to achieve accurate traffic flow prediction results in road networks under bad weather, a novel Traffic-Weather Generative Adversarial Network (TWeather-GAN model) is developed. This model comprises a Generator and a Discriminator. The Generator incorporates both the traffic and weather modules to extract the spatiotemporal patterns hidden in traffic flow and weather data. In the traffic and weather modules, the gated convolutional layer, Encoder-Decoder architecture, and attention mechanism are established. In the Discriminator, the gated convolutional layer and bidirectional long short-term memory neural network are introduced. Traffic flow data under fog, strong wind, and heavy rain are selected to test the seven baseline models and the proposed model, and the ablation experiments are conducted to analyze the mechanism of the proposed model. The experiments demonstrate that the TWeather-GAN model outperforms the baseline models under bad weather, and makes the prediction error have an average reduction of 0.20%–34.81%, 3.21%–35.22%, and 9.46%–39.10%, respectively, under one-step prediction, three-step prediction, and six-step prediction. Furthermore, establishing the gated convolutional layer and the weather module enhances the accuracy of traffic flow prediction under bad weather. Results show that traffic flow fluctuations and distributions differ under fog and strong wind, and heavy rain affects the trend of traffic flow over one day.

TBiSeg: A transformer-based network with bi-level routing attention for inland waterway segmentation

Unmanned surface vehicles (USVs) for inland waterways have recently attracted increasing attention in various fields. Accurate detection in navigable regions is crucial for ensuring USV safety in autonomous navigation. However, the complex and variable environment of inland waterways, such as confusable textures and irregular edge details, continues to pose some problems in existing methods. Therefore, to acquire navigable regions, this study proposed TBiSeg, a Vision Transformer-based efficient inland waterway segmentation network, for obtaining pixel-level results. Bi-level routing attention is used to improve the Transformer block, which enhances the understanding of inland water textures. Additionly, this study combined global and local attention through a hierarchical encoder–decoder architecture. To simulate inland waterway scenes as accurately as possible, this study used two representative public datasets for data integration and data augmentation, and conducted testing and cross-validating using multiple inland waterway datasets. Results demonstrated that the model performed better than current state-of-the-art models in segmentation accuracy and robustness in complex inland waterway environments while showing impressive generalization. The datasets and code used in this paper is available at https://github.com/dawnnazzz/TBiSeg.

GCN-Enhanced Spatial-Spectral Dual-Encoder Network for Simultaneous Segmentation of Retinal Layers and Fluid in OCT Images

Changes in retinal thickness and the morphology of fluid provide valuable diagnostic information for Macular Edema (ME). Optical Coherence Tomography (OCT) is a pivotal diagnostic tool for ME, and the accurate segmentation of OCT retinal images aids ophthalmologists in monitoring disease progression. However, the unpredictable nature of ME’s occurrence poses significant challenges for existing methods in achieving precise segmentation of both retinal layers and fluids simultaneously. In this study, a novel U-shaped encoder-decoder architecture named GD-Net, is introduced for simultaneously segmenting retinal layers of diverse thicknesses and fluids. Specifically, a novel Fast Fourier Encoder is integrated within the encoder branch to capture spectral domain features overlooked by the spatial encoder. Additionally, Multi-scale Graph Convolution module is inserted between the encoder and decoder branches to leverage retinal layer topology for spatial reasoning. GD-Net is further optimized using a joint loss function, which includes a weighted version of region segmentation loss and contour loss, effectively mitigating the issues of blurred fluid edges and internal holes. The method was validated on three publicly available datasets: DUKE DME, Peripapillary OCT, and RETOUCH, achieving overall mean Dice score of 0.839, 0.826, and 0.872, respectively, outperforming the other state-of-the-art techniques. GD-Net demonstrated consistent high-precision segmentation on B-scans from different devices, proving its robustness and surpassing other methods in terms of visual quality, thus offering substantial support for clinical ophthalmologists in diagnosing ME. The code and weights are publicly available at: https://github.com/DBook111/GD-Net.

Kemeny Constant-Based Optimization of Network Clustering Using Graph Neural Networks.

The recent trend in using network and graph structures to represent a variety of different data types has renewed interest in the graph partitioning (GP) problem. This interest stems from the need for general methods that can both efficiently identify network communities and reduce the dimensionality of large graphs while satisfying various application-specific criteria. Traditional clustering algorithms often struggle to capture the complex relationships within graphs and generalize to arbitrary clustering criteria. The emergence of graph neural networks (GNNs) as a powerful framework for learning representations of graph data provides new approaches to solving the problem. Previous work has shown GNNs to be capable of proposing partitionings using a variety of criteria. However, these approaches have not yet been extended to Markov chains or kinetic networks. These arise frequently in the study of molecular systems and are of particular interest to the biomolecular modeling community. In this work, we propose several GNN-based architectures to tackle the GP problem for Markov Chains described as kinetic networks. This approach aims to maximize the Kemeny constant, which is a variational quantity and it represents the sum of time scales of the system. We propose using an encoder-decoder architecture and show how simple GraphSAGE-based GNNs with linear layers can outperform much larger and more expressive attention-based models in this context. As a proof of concept, we first demonstrate the method's ability to cluster randomly connected graphs. We also use a linear chain architecture corresponding to a 1D free energy profile as our kinetic network. Subsequently, we demonstrate the effectiveness of our method through experiments on a data set derived from molecular dynamics. We compare the performance of our method to other partitioning techniques, such as PCCA+. We explore the importance of feature and hyperparameter selection and propose a general strategy for large-scale parallel training of GNNs for discovering optimal graph partitionings.

SKT-IDS: Unknown attack detection method based on Sigmoid Kernel Transformation and encoder–decoder architecture

Intrusion Detection Systems (IDS) are crucial in cybersecurity for monitoring network traffic and identifying potential attacks. Existing IDS research largely focuses on known attack detection, leaving a significant gap in research regarding unknown attack detection, where achieving a balance between false alarm rate (identifying normal traffic as attack traffic) and recall rate of unknown attack detection remains challenging. To address these gaps, we propose a novel IDS based on Sigmoid Kernel Transformation and Encoder-Decoder architecture, namely SKT-IDS, where SKT stands for Sigmoid Kernel Transformation. We start with pre-training an attention-based encoder for coarse-grained intrusion detection. Then, we use this encoder to build an encoder–decoder model specifically for 0-day attack detection, training it solely on known traffic using the cosine similarity loss function. To enhance detection, we introduce a Sigmoid Kernel Transformation for feature engineering, improving the discriminative ability between normal traffic and 0-day attacks. Finally, we conducted a series of ablation and comparative experiments on the NSL-KDD and CSE-CIC-IDS2018 datasets, confirming the effectiveness of our proposed method. With a false alarm rate of 1%, we achieved recall rates for unknown attack detection of 65% and 69% on the two datasets, respectively, demonstrating significant performance improvements compared to existing state-of-the-art models.

Understanding the natural language of DNA using encoder-decoder foundation models with byte-level precision.

This article presents the Ensemble Nucleotide Byte-level Encoder-Decoder (ENBED) foundation model, analyzing DNA sequences at byte-level precision with an encoder-decoder Transformer architecture. ENBED uses a subquadratic implementation of attention to develop an efficient model capable of sequence-to-sequence transformations, generalizing previous genomic models with encoder-only or decoder-only architectures. We use Masked Language Modeling to pretrain the foundation model using reference genome sequences and apply it in the following downstream tasks: (i) identification of enhancers, promotors, and splice sites, (ii) recognition of sequences containing base call mismatches and insertion/deletion errors, an advantage over tokenization schemes involving multiple base pairs, which lose the ability to analyze with byte-level precision, (iii) identification of biological function annotations of genomic sequences, and (iv) generating mutations of the Influenza virus using the encoder-decoder architecture and validating them against real-world observations. In each of these tasks, we demonstrate significant improvement as compared to the existing state-of-the-art results. The source code used to develop and fine-tune the foundation model has been released on Github (https://github.itap.purdue.edu/Clan-labs/ENBED).