Decoder Part Research Articles

MVAD HAN: A Multi-View Based Anomaly Detection Method for Heterogeneous Attributed Networks

With the frequent occurrence of network security incidents in recent years, it has become very important to detect anomalous behaviour in networks as early and accurately as possible. Anomaly detection can improve the security of complex network systems by detecting abnormal and unreliable nodes, and thus it has become a hot research direction that has attracted wide attention. At present, abstracting real complex systems into complex networks for anomaly detection is the mainstream research method. However, the existing methods still have challenges in extracting network heterogeneity information and attribute information, so we propose a multi-view based anomaly detection method for heterogeneous attributed networks, MVAD HAN. This method can better extract the heterogeneous structural information and rich attribute information of the network to model heterogeneous attributed networks. Our method adopts an encoder-decoder architecture. First, in the encoder part, we use the Heterogeneous Graph Transformer with multiple views to learn node embeddings that fuse the heterogeneous information of the network. In the decoder part, we use an inner product decoder to reconstruct the network topology, a multilayer perceptron-based decoder to better reconstruct the network attribute information, and a linear projection to reconstruct the node type information of the network. Finally, we compute an anomaly score for each node using three reconstruction errors: network structure, attributes and node type. The higher the reconstruction error of a node, the higher the anomaly score and the higher the probability of an anomaly. Finally, anomalous nodes are identified by ranking the anomaly scores and setting a threshold. We validate the effectiveness of the proposed method on four real-world datasets. The experimental results show that this method outperforms several of the baseline methods and has a good performance in anomaly detection.

PCL–PTD Net: Parallel Cross-Learning-Based Pixel Transferred Deconvolutional Network for Building Extraction in Dense Building Areas With Shadow

Urban building segmentation from remote sensed imageries is challenging because there usually exists a variety of building features. Furthermore, very high spatial resolution imagery can provide many details of the urban building, such as styles, small gaps among buildings, building shadows, etc. Hence, satisfactory accuracy in detecting and extracting urban features from highly detailed images still remains. Deep learning semantic segmentation using baseline networks works well on building extraction; however, their ability in building extraction in shadows area, unclear building feature, and narrow gaps among buildings in dense building zone is still limited. In this article, we propose parallel cross-learning-based pixel transferred deconvolutional network (PCL–PTD net), and then is used to segment urban buildings from aerial photographs. The proposed method is evaluated and intercompared with traditional baseline networks. In PCL–PTD net, it is composed of parallel network, cross-learning functions, residual unit in encoder part, and PTD in decoder part. The performance is applied to three datasets (Inria aerial dataset, international society for photogrammetry and remote sensing Potsdam dataset, and UAV building dataset), to evaluate its accuracy and robustness. As a result, we found that PCL–PTD net can improve learning capacities of the supervised learning model in differentiating buildings in dense area and extracting buildings covered by shadows. As compared to the baseline networks, we found that proposed network shows superior performance compared to all eight networks (SegNet, U-net, pyramid scene parsing network, PixelDCL, DeeplabV3+, U-Net++, context feature enhancement networ, and improved ResU-Net). The experiments on three datasets also demonstrate the ability of proposed framework and indicating its performance.

Real-Time Multiple Guava Leaf Disease Detection from a Single Leaf Using Hybrid Deep Learning Technique

The guava plant has achieved viable significance in subtropics and tropics owing to its flexibility to climatic environments, soil conditions and higher human consumption. It is cultivated in vast areas of Asian and Non-Asian countries, including Pakistan. The guava plant is vulnerable to diseases, specifically the leaves and fruit, which result in massive crop and profitability losses. The existing plant leaf disease detection techniques can detect only one disease from a leaf. However, a single leaf may contain symptoms of multiple diseases. This study has proposed a hybrid deep learning-based framework for the real-time detection of multiple diseases from a single guava leaf in several steps. Firstly, Guava Infected Patches Modified MobileNetV2 and U-Net (GIP-MU-NET) has been proposed to segment the infected guava patches. The proposed model consists of modified MobileNetv2 as an encoder, and the U-Net model’s up-sampling layers are used as a decoder part. Secondly, the Guava Leaf Segmentation Model (GLSM) is proposed to segment the healthy and infected leaves. In the final step, the Guava Multiple Leaf Diseases Detection (GMLDD) model based on the YOLOv5 model detects various diseases from a guava leaf. Two self-collected datasets (the Guava Patches Dataset and the Guava Leaf Diseases Dataset) are used for training and validation. The proposed method detected the various defects, including five distinct classes, i.e., anthracnose, insect attack, nutrition deficiency, wilt, and healthy. On average, the GIP-MU-Net model achieved 92.41% accuracy, the GLSM gained 83.40% accuracy, whereas the proposed GMLDD technique achieved 73.3% precision, 73.1% recall, 71.0% mAP@0.5 and 50.3 mAP@0.5:0.95 scores for all the aforesaid classes.

AGD-Linknet: A Road Semantic Segmentation Model for High Resolution Remote Sensing Images Integrating Attention Mechanism, Gated Decoding Block and Dilated Convolution

Road information is a kind of important geographic information. Road information extracted from remote sensing images is widely used in map, traffic, navigation, military and many other fields. However, the autonomous extraction of road information from high resolution remote sensing images has some problems such as incoherence, incompleteness and poor connectivity, therefore, a semantic segmentation model for roads in high resolution remote sensing images, called AGD-Linknet, is proposed, which integrates attention mechanisms, gated decoder block, and dilated convolution. This model mainly consists of three parts.Firstly, the stem block is used as the initial convolution layer of the model to reduce the information loss in the convolution stage; Secondly, the series-parallel combined dilated convolution and coordinate attention block into the center of the network, which enlarges the receptive field of the network and improves the feature extraction ability of spatial domain and channel domain information; Finally, in the decoder part, gated convolution is introduced in the decoder part to improve the extraction of road edge. Compared with U-Net, Linknet and D-Linknet on the DeepGlobe dataset, the proposed AGD-Linknet has improved the pixel accuracy, mean intersection over union and F1-Score index of road recognition by 1.41%-11.52%, 0.0077-0.1473, 0.0057-0.1292, and has certain effectiveness and feasibility in many scenarios in rural areas, urban, and suburbs. And can be apply to the tasks of road recognition and extraction in high-resolution remote sensing.

Oceanic Eddy Identification Using Pyramid Split Attention U-Net With Remote Sensing Imagery

Oceanic eddy is the ubiquitous ocean flow phenomenon, which has been the key factor in the transportation of ocean energy and materials. Consequently, oceanographic understanding can be enhanced by the intelligent identification of eddy. The state-of-the-art deep learning technologies are gradually improving the identification methods. This letter proposes the pyramid split attention(PSA) eddy detection U-Net architecture (PSA-EDUNet) that targets oceanic eddy identification from ocean remote sensing imagery. As for the PSA-EDUNet, its inspiration comes from U-Net, which contains encoder and decoder parts, making the integration of inferior and senior features efficiently and ensuring the feature information will not be lost in large quantities through non-linear connection mode. Meanwhile, PAS module is introduced to enhance feature extraction. In terms of the fusion data, the sea surface feature is the main criterion of eddy identification, including sea surface temperature (SST) and sea level anomaly (SLA). The experiments are implemented on the Kuroshio Extension and the South Atlantic regions, the results demonstrate that the proposed method can outperform other methods, especially for eddy edges and small-scale eddies.

A Text-Guided Generation and Refinement Model for Image Captioning

A high-quality image description requires not only the logic and fluency of language but also the richness and accuracy of the content. However, due to the semantic gap between vision and language, most existing image captioning approaches that directly learn the cross-modal mapping from the vision to language are difficult to meet these two requirements simultaneously. Inspired by the progressive learning mechanism, we trace the generating + refining route and propose a novel Text-Guided Generation and Refinement (dubbed as TGGAR) model with assistance from the guide text to improve the quality of captions. The guide text is selected from the training set according to the visual content similarity, then utilized to explore the salient objects and extend the candidate words. Specifically, we follow the encoder-decoder architecture, and design a Text-Guided Relation Encoder (TGRE) to learn the visual representation that is more consistent with human visual cognition. Besides, we divide the decoder part into two submodules: a Generator for the primary sentence generation and a Refiner for the sentence refinement. Generator, consisting of a standard LSTM and a Gate on Attention (GOA) module, aims to generate the primary sentence logically and fluently. Refiner contains a caption encoder module, an attention-based LSTM and a GOA module, which iteratively modifies the details in the primary caption to make captions rich and accurate with help of the guide text. Extensive experiments on the MS COCO captioning dataset demonstrate that our framework with fewer parameters remains comparable to transformer-based methods, and achieves state-of-the-art performance compared with other relevant approaches.

CFNet: An Eigenvalue Preserved Approach to Multiscale Building Segmentation in High-Resolution Remote Sensing Images

In recent years, AI and Deep Learning (DL) methods have been widely used for object classification, recognition, and segmentation of high-resolution multispectral remote sensing images. These DL-based solutions perform better compare to traditional spectral algorithms but still suffer from insufficient optimization of global and local features of object context. In addition, failure of code-data isolation and/or disclosure of detailed eigenvalues causes serious privacy and even secret leakage due to the sensitivity of high-resolution remote sensing data and their processing mechanisms. In this paper, Class Feature (CF) modules have been presented in the decoder part of an attention-based CNN network to distinguish between building and non-building (background) area. In this way, context features of a focused object can be extracted with more details being processed, whilst the resolution of images is maintained. The reconstructed local and global feature values and dependencies in the proposed model are maintained by reconfiguring multiple effective attention modules with contextual dependencies to achieve better results for the eigenvalue. According to quantitative results and their visualization, the proposed model has depicted better performance over others' work using two large-scale building remote sensing datasets. The F1-score of this model reached 87.91 and 89.58 on WHU Buildings Dataset and Massachusetts Buildings Dataset, respectively, which exceeded the other semantic segmentation models.

GNet: An integrated context-aware neural framework for transcription factor binding signal at single nucleotide resolution prediction.

Transcription factors (TFs) are important factors that regulate gene expression. Revealing the mechanism affecting the binding specificity of TFs is the key to understanding gene regulation. Most of the previous studies focus on TF-DNA binding sites at the sequence level, and they seldom utilize the contextual features of DNA sequences. In this paper, we develop an integrated spatiotemporal context-aware neural network framework, named GNet, for predicting TF-DNA binding signal at single nucleotide resolution by achieving three tasks: single nucleotide resolution signal prediction, identification of binding regions at the sequence level, and TF-DNA binding motif prediction. GNet extracts implicit spatial contextual information with a gated highway neural mechanism, which captures large context multi-level patterns using linear shortcut connections, and the idea of it permeates the encoder and decoder parts of GNet. The improved dual external attention mechanism, which learns implicit relationships both within and among samples, and improves the performance of the model. Experimental results on 53 human TF ChIP-seq datasets and 6 chromatin accessibility ATAC-seq datasets shows that GNet outperforms the state-of-the-art methods in the three tasks, and the results of cross-species studies on 15 human and 18 mouse TF datasets of the corresponding TF families indicate that GNet also shows the best performance in cross-species prediction over the competitive methods.

Unsupervised learning with a physics-based autoencoder for estimating the thickness and mixing ratio of pigments.

Layered surface objects represented by decorated tomb murals and watercolors are in danger of deterioration and damage. To address these dangers, it is necessary to analyze the pigments' thickness and mixing ratio and record the current status. This paper proposes an unsupervised autoencoder model for thickness and mixing ratio estimation. The input of our autoencoder is spectral data of layered surface objects. Our autoencoder is unique, to our knowledge, in that the decoder part uses a physical model, the Kubelka-Munk model. Since we use the Kubelka-Munk model for the decoder, latent variables in the middle layer can be interpretable as the pigment thickness and mixing ratio. We conducted a quantitative evaluation using synthetic data and confirmed that our autoencoder provides a highly accurate estimation. We measured an object with layered surface pigments for qualitative evaluation and confirmed that our method is valid in an actual environment. We also present the superiority of our unsupervised autoencoder over supervised learning.

Transformer-Based Long Distance Fiber Channel Modeling for Optical OFDM Systems

The fiber channel model plays an essential role in the simulation and design of optical fiber communication systems. However, it is difficult for conventional model-driven modeling to balance accuracy and efficiency, especially in optical orthogonal frequency division multiplexing (OFDM) systems with complex and long-haul transmission. We introduce the simplified Transformer into optical OFDM systems and combine it with the feature decoupled distributed (FDD) scheme for fast and accurate fiber channel modeling. Unlike the popular Transformer architectures, we remove the Decoder part and cancel the self-attention with quadratic complexity, significantly reducing the computational cost. The modeling performance is investigated from the nonlinear fitting capability, accuracy, and generalization ability. The transmission distance ranges from 80 km to 1600 km. The highly matched four-wave mixing (FWM) power, low error vector magnitudes (EVMs), and similar signal-noise ratios (SNRs) demonstrate the high precision and robustness of the model. Furthermore, the modeling is studied under different transmission rates and is proved to be reliable over a wide transmission bandwidth. Compared to the bidirectional long short-term memory (Bi-LSTM), the Transformer performs better in accuracy and has lower computational and memory costs. For modeling under the same conditions, the required running time of the Transformer is about 60% of Bi-LSTM, less than 1% of the split-step Fourier method (SSFM). The Transformer-based method achieves high precision modeling of the fiber channel in the long-distance and high-rate optical OFDM system and makes a significant breakthrough in complexity.

SegMarsViT: Lightweight Mars Terrain Segmentation Network for Autonomous Driving in Planetary Exploration

Planetary rover systems need to perform terrain segmentation to identify feasible driving areas and surround obstacles, which falls into the research area of semantic segmentation. Recently, deep learning (DL)-based methods were proposed and achieved great performance for semantic segmentation. However, due to the on-board processor platform’s strict comstraints on computational complexity and power consumption, existing DL approaches are almost impossible to be deployed on satellites under the burden of extensive computation and large model size. To fill this gap, this paper targeted studying effective and efficient Martian terrain segmentation solutions that are suitable for on-board satellites. In this article, we propose a lightweight ViT-based terrain segmentation method, namely, SegMarsViT. In the encoder part, the mobile vision transformer (MViT) block in the backbone extracts local–global spatial and captures multiscale contextual information concurrently. In the decoder part, the cross-scale feature fusion modules (CFF) further integrate hierarchical context information and the compact feature aggregation module (CFA) combines multi-level feature representation. Moreover, we evaluate the proposed method on three public datasets: AI4Mars, MSL-Seg, and S5Mars. Extensive experiments demonstrate that the proposed SegMarsViT was able to achieve 68.4%, 78.22%, and 67.28% mIoU on the AI4Mars-MSL, MSL-Seg, and S5Mars, respectively, under the speed of 69.52 FPS.

Attention-Based Sequence-to-Sequence Model for Time Series Imputation.

Time series data are usually characterized by having missing values, high dimensionality, and large data volume. To solve the problem of high-dimensional time series with missing values, this paper proposes an attention-based sequence-to-sequence model to imputation missing values in time series (ASSM), which is a sequence-to-sequence model based on the combination of feature learning and data computation. The model consists of two parts, encoder and decoder. The encoder part is a BIGRU recurrent neural network and incorporates a self-attentive mechanism to make the model more capable of handling long-range time series; The decoder part is a GRU recurrent neural network and incorporates a cross-attentive mechanism into associate with the encoder part. The relationship weights between the generated sequences in the decoder part and the known sequences in the encoder part are calculated to achieve the purpose of focusing on the sequences with a high degree of correlation. In this paper, we conduct comparison experiments with four evaluation metrics and six models on four real datasets. The experimental results show that the model proposed in this paper outperforms the six comparative missing value interpolation algorithms.

Coherence-aware context aggregator for fast video object segmentation

Semi-supervised video object segmentation (VOS) is a highly challenging problem that has attracted much research attention in recent years. Temporal context plays an important role in VOS by providing object clues from the past frames. However, most of the prevailing methods directly use the predicted temporal results to guide the segmentation of the current frame, while ignoring the coherence of temporal context, which may be misleading and degrade the performance. In this paper, we propose a novel model named Coherence-aware Context Aggregator (CCA) for VOS, which consists of three modules. First, a coherence-aware module (CAM) is proposed to evaluate the coherence of the predicted result of the current frame and then fuses the coherent features to update the temporal context. CAM can determine whether the prediction is accurate, thus guiding the update of the temporal context and avoiding the introduction of erroneous information. Second, we devise a spatio-temporal context aggregation (STCA) module to aggregate the temporal context with the spatial feature of the current frame to learn a robust and discriminative target representation in the decoder part. Third, we design a refinement module to refine the coarse feature generated from the STCA module for more precise segmentation. Additionally, CCA uses a cropping strategy and takes small-size images as input, thus making it computationally efficient and achieving a real-time running speed. Extensive experiments on four challenging benchmarks show that CCA achieves a better trade-off between efficiency and accuracy compared to state-of-the-art methods. The code will be public.

Spatiotemporal model based on transformer for bias correction and temporal downscaling of forecasts

Numerical weather prediction (NWP) provides the future state of the atmosphere and is a major tool for weather forecasting. However, NWP has inevitable errors and requires bias correction to obtain more accurate forecasts. NWP is based on discrete numerical calculations, which inevitably result in a loss in resolution, and downscaling provides important support for obtaining detailed weather forecasts. In this paper, based on the spatio-temporal modeling approach, the Spatio-Temporal Transformer U-Net (ST-UNet) is constructed based on the U-net framework using the swin transformer and convolution to perform bias correction and temporal downscaling. The encoder part extracts features from the multi-time forecasts, and the decoder part uses the features from the encoder part and the constructed query vector for feature reconstruction. Besides, the query builder block generates different query vectors to accomplish different tasks. Multi-time bias correction was conducted for the 2-m temperature and the 10-m wind component. The results showed that the deep learning model significantly outperformed the anomaly numerical correction with observations, and ST-UNet also outperformed the U-Net model for single-time bias correction and the 3-dimensional U-Net (3D-UNet) model for multi-time bias correction. Forecasts from ST-UNet obtained the smallest root mean square error and the largest accuracy and correlation coefficient on both the 2-m temperature and 10-m wind component experiments. Meanwhile, temporal downscaling was performed to obtain hourly forecasts based on ST-UNet, which increased the temporal resolution and reduced the root mean square error by 0.78 compared to the original forecasts. Therefore, our proposed model can be applied to both bias correction and temporal downscaling tasks and achieve good accuracy.

MFENet: Multitype fusion and enhancement network for detecting salient objects in RGB-T images

Recent progress in salient object detection (SOD) has been fueled substantially by the development of convolutional neural networks. However, several SOD methods do not fully exploit information from different modalities, consequently performing only marginally better than methods using a single modality. Therefore, we propose a multitype fusion and enhancement network (MFENet), following three steps “Encoder- Pre-decoder- Decoder” for RGB-thermal (RGB-T) SOD by completely exploiting the advantages of the RGB and thermal modalities through feature integration and enhancement. To better fuse two modalities' features, we have designed the cross-modality fusion module (CMFM) in the encoder part. As shallow features describe details and deep features provide semantic information, a multiscale interactive refinement module is designed in the pre-decoder part to complement multilevel features. Additionally, to further sharpen salient objects, we have proposed a high-level, low-level module that takes inputs from adjacent layers for gradual translation into a saliency map in the decoder part. This module provides semantic information for shallower features and the boundaries of salient objects can be gradually sharpened with subtle details. Extensive experiments show the effectiveness and robustness of the proposed MFENet and its substantial improvement over state-of-the-art RGB-T SOD methods. The codes and results will be available at: https://github.com/wujunyi1412/MFENet_DSP.

Transformer-Based Maneuvering Target Tracking.

When tracking maneuvering targets, recurrent neural networks (RNNs), especially long short-term memory (LSTM) networks, are widely applied to sequentially capture the motion states of targets from observations. However, LSTMs can only extract features of trajectories stepwise; thus, their modeling of maneuvering motion lacks globality. Meanwhile, trajectory datasets are often generated within a large, but fixed distance range. Therefore, the uncertainty of the initial position of targets increases the complexity of network training, and the fixed distance range reduces the generalization of the network to trajectories outside the dataset. In this study, we propose a transformer-based network (TBN) that consists of an encoder part (transformer layers) and a decoder part (one-dimensional convolutional layers), to track maneuvering targets. Assisted by the attention mechanism of the transformer network, the TBN can capture the long short-term dependencies of target states from a global perspective. Moreover, we propose a center-max normalization to reduce the complexity of TBN training and improve its generalization. The experimental results show that our proposed methods outperform the LSTM-based tracking network.

Prior Guided Transformer for Accurate Radiology Reports Generation.

In this paper, we propose a prior guided transformer for accurate radiology reports generation. In the encoder part, a radiograph is firstly represented by a set of patch features, which is obtained through a convolutional neural network and a traditional transformer encoder. Then an Additive Gaussian model is applied to represent the prior knowledge based on unsupervised clustering and sparse attention. In the decoder part, prior embeddings are acquired by probabilistically sampling from the radiograph prior. Then the visual features, language embeddings, and prior embeddings are fused by our proposed Prior Guided Attention to generate accurate radiology reports. Experiment results show that our method achieves better performance than state-of-the-art methods on two public radiology datasets, which proves the effectiveness of our prior guided transformer.

Attention augmented residual autoencoder for efficient polyp segmentation

AbstractColon cancer has been reported to be one of the frequently diagnosed cancers and the leading cause of cancer deaths. Early detection and removal of malicious polyps, which are precursors of colon cancer, can enormously lessen the fatality rate. The detection and segmentation of polyps in colonoscopy is a challenging task even for an experienced colonoscopist, due to divergences in the size, shape, texture, and the close resemblance of polyps with the colon lining. Machine‐assisted detection, localization, and segmentation of polyps in the screening procedure can profoundly help the clinicians. Autoencoder‐based architectures used in polyp segmentation lack the efficiency in incorporating both local and long‐range pixel dependencies. To address the challenges in the automatic segmentation of colon polyps we propose an autoencoder architecture, augmented with a feature attention module in the decoder part. The salient features from RGB colonoscopic images are extracted using the residual skip‐connected autoencoder. The decoder attention module joins spatial subspace with feature subspace extracted from the deep residual convolutional neural network and enhances the feature weight for precise segmentation of polyp regions. Extensive experiments on four publicly available polyp datasets demonstrate that the proposed architecture provides very impressive performance in terms of segmentation metrics (Dice scores and Jaccard scores) when compared with the state‐of‐the‐art polyp segmentation approaches.

Compact representation and identification of important regions of metal microstructures using complex-step convolutional autoencoders

In this study, we propose a complex-step convolutional autoencoder to identify the regions that are important in a metal microstructure for compact representation and secure sharing. Firstly, the architecture of a convolutional autoencoder is designed for the compact representation of microstructural images. The designed autoencoder achieved a high image compression ratio of 32 without loss of important information. Secondly, an in-home developed model agnostic sensitivity analysis using complex step derivative approximation is implemented on convolutional autoencoders to identify regions of the microstructure that are important for reconstruction. Finally, saliency maps that highlight the importance of pixels for reconstruction are generated for three grades of dual-phase structural steels. The saliency maps indicated secondary phase regions and grain boundaries are important for microstructure image reconstruction. The proposed approach produces more tenable saliency explanations compared to guided backpropagation and layer wise relevance propagation methods. The decoder part of the convolutional autoencoder can be used as a key that could be used to reconstruct the actual microstructure from encoded image information contributing to secure and efficient sharing of microstructure data. The proposed framework is generic and can be extended to identify important microstructural regions for other metals, composites, biomaterials, and material systems.

Pixel-Wise Ship Identification From Maritime Images via a Semantic Segmentation Model

Accurately identifying ships from maritime surveillance videos attracts increasing attention in the smart ship community, considering that the videos provide informative yet easily understandable spatial–temporal traffic information for varied maritime traffic participants. Previous studies (e.g., ship detection and ship tracking) are conducted by learning distinct features from training samples labeled in terms of bounding boxes, and thus, background pixels may be wrongly trained as ship features. To bridge the gap, we propose a novel approach for achieving a pixel-wise ship segmentation and identification task through a novel design of U-Net deep learning architecture (denoted as EU-Net). The encoder of the EU-Net extracts distinct ship features from input maritime images, and its decoder outputs ship segmentation results in the pixel-wise manner. The proposed EU-Net model consists of encoder and decoder parts via the help of a convolution layer, a depth separable convolution layer, a softmax layer, and so on. More specifically, the EU-Net model identifies each pixel into ship or non-ship as the final output. Experimental results suggest that our proposed model can accurately identify ship (in terms of pixels), considering that the ship segmentation accuracies were larger than 99%. The proposed ship segmentation framework can be further adaptively deployed in the ship sensing system for maritime traffic situation awareness and intelligent visual navigation in a smart ship era.