Encoder-decoder Network Research Articles

Lightweight multi-scale encoder– decoder network with locally enhanced attention mechanism for concrete crack segmentation

Abstract Concrete surface crack detection and maintenance are crucial for ensuring structural safety. Deep learning-based techniques for detecting concrete cracks have become popular due to the quick advancement of artificial intelligence. However, the actual uses of these methods are limited due to issues like large model sizes and significant dependence on powerful computing hardware. To address these issues, this paper presents a lightweight multi-scale encoder-decoder network (LMED-Net) for crack detection of concrete structures. LMED-Net employs MobileNetV2 as the encoder for the initial feature extraction. A multi-scale feature extraction (MFE) module is developed and serially attached after the encoder for refining feature extraction. Finally, to strengthen the network's perception of pixels surrounding the cracks, a novel enhanced attention mechanism (EAM) is deployed in the decoder. By improving the network's attention to information within the crack regions, this mechanism keeps contextual information from being lost. Comparative experimental results show that the proposed network achieves an F1 Score(F1) of 60.32% and a mean intersection over union (mIoU) of 71.04% on the CFD dataset. On the DeepCrack dataset, the F1 and mIoU increase to 79.09% and 81.85% respectively. Notably, LMED-Net performs exceptionally well in crack segmentation since its model size and parameters count are much smaller than those of other image segmentation methods. Furthermore, ablation studies further validate the effectiveness of the proposed MFE module and EAM.

Industrial multivariate time-series data anomaly detection incorporating attention mechanisms and adversarial training

ABSTRACT To address the challenges faced in industrial anomaly detection, including data sample imbalance, lack of anomaly labels, and complex spatiotemporal relationships in high-dimensional data, this paper proposes a novel multi-modal time-series anomaly detection model that combines attention mechanisms and adversarial training. In this model, the first step involves utilizing graph attention mechanisms to extract sequence correlation features from multi-modal time-series data, which are then summed with the original data to form a dual-feature-based data representation. Subsequently, a self-supervised learning approach is employed to input this data representation into a variational autoencoder’s encoding-decoding network for reconstruction. Anomaly detection is performed by analyzing the error between the input and reconstructed data. The model also employs spatiotemporal attention mechanisms and adversarial training during reconstruction to enhance feature extraction and model generalization. By comparing our proposed model to five commonly used baseline models, we demonstrate its effectiveness in detecting anomalies in scenarios involving high-dimensional data and imbalanced abnormal samples, demonstrating superior anomaly detection performance, as well as excellent performance on real industrial production and processing datasets.

Triple-attentions based salient object detector for strip steel surface defects

Accurate detection of surface defects on strip steel is essential for ensuring strip steel product quality. Existing deep learning based detectors for strip steel surface defects typically strive to iteratively refine and integrate the coarse outputs of the backbone network, enhancing the models’ ability to express defect characteristics. Attention mechanisms including spatial attention, channel attention and self-attention are among the most prevalent techniques for feature extraction and fusion. This paper introduces an innovative triple-attention mechanism (TA), characterized by interrelated and complementary interactions, that concurrently and iteratively refines and integrates feature maps from three distinct perspectives, thereby enhancing the features’ capacity for representation. The idea is from the following observation: given a three-dimensional feature map, we can examine the feature map from the three different yet interrelated two-dimensional planar perspectives: the channel-width, channel-height, and width-height perspectives. Based on the TA, a novel detector, called TADet, for the detection of steel strip surface defects is proposed, which is an encoder-decoder network: the decoder uses the proposed TA refines/fuses the multiscale rough features generated by the encoder (backbone network) from the three distinct perspectives (branches) and then integrates the purified feature maps from the three branches. Extensive experimental results show that TADet is superior to the state-of-the-art methods in terms of mean absolute error, S-measure, E-measure and F-measure, confirming the effectiveness and robustness of the proposed TADet. Our code and experimental results are available at https://github.com/hpguo1982/TADet.

Predictive modelling of seismic wave fields: learning the transfer function using encoder–decoder networks

SUMMARY Wouldn't it be beneficial if we could predict the time-series at a seismic station even if the station no longer exists? In geophysical data analysis, this capability would enhance our ability to study and monitor seismic events and seismic noise, particularly in regions with incomplete station coverage or where stations are temporarily offline. This study introduces a novel adaption of encoder–decoder networks from the subfield of deep learning, modified to predict the development of seismic wave fields between two seismic stations. Using 1-D time-series measurements, our algorithm aims to learn and predict signal transformations between the two stations by approximating the transfer function. Initially, we evaluate this proof of concept in a simplified controlled setting using synthetic data, before we incorporate field data gathered at a seismic exploration site in an area containing several roads, wind turbines, oil pump jacks and railway traffic. Across diverse scenarios, the model demonstrates proficiency in learning the transfer function among various seismic station configurations. Particularly, it achieves high accuracy in predicting a majority of seismic wave phases across different data sets. Diverging significantly from encoder–decoder networks that estimate time-series forecasts by analysing historical trends, our approach places greater emphasis on the wave propagation between nearby locations. Thereby, the analysis incorporates both phase and amplitude information and provides a new approach to approximate the transfer function relying on machine learning techniques. The gained knowledge enables to reconstruct data from missing, offline or defunct stations in the context of temporary seismic arrays or exclude non-relevant data for denoising.

Recurrent attention encoder–decoder network for multi-step interval wind power prediction

Recurrent attention encoder–decoder network for multi-step interval wind power prediction

Human motion similarity evaluation based on deep metric learning

In order to eliminate the impact of camera viewpoint factors and human skeleton differences on the action similarity evaluation and to address the issue of human action similarity evaluation under different viewpoints, a method based on deep metric learning is proposed in this article. The method trains an automatic encoder-decoder deep neural network model by means of a homemade synthetic dataset, which maps the 2D human skeletal key point sequence samples extracted from motion videos into three potential low-dimensional dense spaces. Action feature vectors independent of camera viewpoint and human skeleton structure are extracted in the low-dimensional dense spaces, and motion similarity metrics are performed based on these features, thereby effectively eliminating the effects of camera viewpoint and human skeleton size differences on motion similarity evaluation. Specifically, when extracting the action information feature vectors using the automatic encoder-decoder network model, a sliding window method is used to divide the key point sequences of each limb part into sequence patches, and the action information feature vectors independent of the camera viewpoint and skeleton structure are extracted in a smaller time unit, so as to obtain a more refined action similarity evaluation result. In addition, the dynamic time warping (DWT) algorithm is exploited to align the sequence of action information feature vectors temporally, which solves the problem of temporal axis discrepancies in realizing similarity metrics based on action information feature vectors. More accurate and reliable human action similarity evaluation results were achieved by the loss function composed of three components, namely, cross-reconstruction loss, reconstruction loss and triplet loss. Finally, the performance of the algorithm is evaluated in a homemade dataset, and the experimental results show that the method could effectively eliminate the influence of the differences in camera viewpoints and human skeleton sizes on the similarity evaluation of actions, and generate more reliable and closer to the human subjective perception of similarity evaluation results for human actions captured from different viewpoints or with varying skeleton sizes.

Human body contour extraction method based on human skeleton key point guidance

By extracting human contours from 2D images captured by the camera and then obtaining human size data, the cost of garment custom measurement can be effectively reduced and the efficiency of custom measurement can be improved. The extraction of human contours plays an important role in the collection of online human size data. We propose a method to extract human contours by fusing the prior information of human skeleton key points into the salience target detection network. Specifically, the skeleton key point information extracted based on OpenPose is fused into the encoder-decoder network for rough detection of the human body target, and the residual refinement network is used to fine-adjust the human body matting, so as to achieve accurate human contour extraction. In this paper, the accuracy and superiority of the algorithm are verified in the public data set P3M-10K of human body matting and applied to the 2D body measurement WeChat applet on mobile phone and computer website.

Reconstruction of Optical Coherence Tomography Images from Wavelength Space Using Deep Learning.

Conventional Fourier domain Optical Coherence Tomography (FD-OCT) systems depend on resampling into a wavenumber (k) domain to extract the depth profile. This either necessitates additional hardware resources or amplifies the existing computational complexity. Moreover, the OCT images also suffer from speckle noise, due to systemic reliance on low-coherence interferometry. We propose a streamlined and computationally efficient approach based on Deep Learning (DL) which enables reconstructing speckle-reduced OCT images directly from the wavelength (λ) domain. For reconstruction, two encoder-decoder styled networks, namely Spatial Domain Convolution Neural Network (SD-CNN) and Fourier Domain CNN (FD-CNN), are used sequentially. The SD-CNN exploits the highly degraded images obtained by Fourier transforming the (λ) domain fringes to reconstruct the deteriorated morphological structures along with suppression of unwanted noise. The FD-CNN leverages this output to enhance the image quality further by optimization in the Fourier domain (FD). We quantitatively and visually demonstrate the efficacy of the method in obtaining high-quality OCT images. Furthermore, we illustrate the computational complexity reduction by harnessing the power of DL models. We believe that this work lays the framework for further innovations in the realm of OCT image reconstruction.

Determination of spherical coordinates of sampled cosmic ray flux distribution using Principal Components Analysis and deep Encoder-Decoder network

Determination of spherical coordinates of sampled cosmic ray flux distribution using Principal Components Analysis and deep Encoder-Decoder network

GLMAFuse: A Dual-Stream Infrared and Visible Image Fusion Framework Integrating Local and Global Features with Multi-Scale Attention

Integrating infrared and visible-light images facilitates a more comprehensive understanding of scenes by amalgamating dual-sensor data derived from identical environments. Traditional CNN-based fusion techniques are predominantly confined to local feature emphasis due to their inherently limited receptive fields. Conversely, Transformer-based models tend to prioritize global information, which can lead to a deficiency in feature diversity and detail retention. Furthermore, methods reliant on single-scale feature extraction are inadequate for capturing extensive scene information. To address these limitations, this study presents GLMAFuse, an innovative dual-stream encoder–decoder network, which utilizes a multi-scale attention mechanism to harmoniously integrate global and local features. This framework is designed to maximize the extraction of multi-scale features from source images while effectively synthesizing local and global information across all layers. We introduce the global-aware and local embedding (GALE) module to adeptly capture and merge global structural attributes and localized details from infrared and visible imagery via a parallel dual-branch architecture. Additionally, the multi-scale attention fusion (MSAF) module is engineered to optimize attention weights at the channel level, facilitating an enhanced synergy between high-frequency edge details and global backgrounds. This promotes effective interaction and fusion of dual-modal features. Extensive evaluations using standard datasets demonstrate that GLMAFuse surpasses the existing leading methods in both qualitative and quantitative assessments, highlighting its superior capability in infrared and visible image fusion. On the TNO and MSRS datasets, our method achieves outstanding performance across multiple metrics, including EN (7.15, 6.75), SD (46.72, 47.55), SF (12.79, 12.56), MI (2.21, 3.22), SCD (1.75, 1.80), VIF (0.79, 1.08), Qbaf (0.58, 0.71), and SSIM (0.99, 1.00). These results underscore its exceptional proficiency in infrared and visible image fusion.

Multi-modality NDE fusion using encoder–decoder networks for identify multiple neurological disorders from EEG signals

Background The complexity and diversity of brain activity patterns make it difficult to accurately diagnose neurological disorders such epilepsy, Parkinson's disease, schizophrenia, stroke, and Alzheimer's disease. Integrated and effective analysis of multiple data sources is often beyond the scope of traditional diagnostic procedures. With the use of multi-modal data, recent developments in neural network approaches present encouraging opportunities for raising diagnostic accuracy. Objectives A novel approach has been proposed toward the integration of different Nondestructive Evaluation data with EEG signals for improving the diagnosis of neurological disorders such as stroke, epilepsy, Parkinson's disease, and schizophrenia, by leveraging advanced neural network techniques in order to improve the identification and correlation of shared latent features across heterogeneous NDE datasets. Methods We determined the 2D scalogram images using a specific encoder-decoder neural network after transforming the EEG signals using wavelet signal processing. Several NDE data types can be easily integrated for thorough analysis due to this network's ability to extract and correlate important aspects from each form of data. Aiming to uncover common patterns indicating of neurological disorders, the technique was evaluated on datasets containing EEG signals and corresponding NDE data. Results Our method demonstrated a significant improvement in diagnostic accuracy and efficiency. The encoder-decoder network effectively identified shared latent features across the heterogeneous NDE datasets, leading to more precise and reliable diagnoses. The fusion of multi-modality NDE data with EEG signals provided a robust framework for the automatic identification of multiple neurological disorders. Conclusions This innovative approach represents a substantial advancement in the field of neurological disorder diagnosis. By integrating diverse NDE data with EEG signals through advanced neural network techniques, we have developed a method that enhances the accuracy and efficiency of diagnosing multiple neurological conditions. This fusion of multi-modality data has the potential to revolutionize current diagnostic practices in neurology, paving the way for more precise and automated identification of neurological disorders.

A multi-scale attention encoding and dynamic decoding network designed for short-term precipitation forecasting

A multi-scale attention encoding and dynamic decoding network designed for short-term precipitation forecasting

Pneumothorax detection and segmentation from chest X-ray radiographs using a patch-based fully convolutional encoder-decoder network.

Pneumothorax, a life-threatening condition characterized by air accumulation in the pleural cavity, requires early and accurate detection for optimal patient outcomes. Chest X-ray radiographs are a common diagnostic tool due to their speed and affordability. However, detecting pneumothorax can be challenging for radiologists because the sole visual indicator is often a thin displaced pleural line. This research explores deep learning techniques to automate and improve the detection and segmentation of pneumothorax from chest X-ray radiographs. We propose a novel architecture that combines the advantages of fully convolutional neural networks (FCNNs) and Vision Transformers (ViTs) while using only convolutional modules to avoid the quadratic complexity of ViT's self-attention mechanism. This architecture utilizes a patch-based encoder-decoder structure with skip connections to effectively combine high-level and low-level features. Compared to prior research and baseline FCNNs, our model demonstrates significantly higher accuracy in detection and segmentation while maintaining computational efficiency. This is evident on two datasets: (1) the SIIM-ACR Pneumothorax Segmentation dataset and (2) a novel dataset we curated from The Medical City, a private hospital in the Philippines. Ablation studies further reveal that using a mixed Tversky and Focal loss function significantly improves performance compared to using solely the Tversky loss. Our findings suggest our model has the potential to improve diagnostic accuracy and efficiency in pneumothorax detection, potentially aiding radiologists in clinical settings.

Enhanced Panoramic Radiograph-Based Tooth Segmentation and Identification Using an Attention Gate-Based Encoder-Decoder Network.

Dental disorders are one of the most important health problems, affecting billions of people all over the world. Early diagnosis is important for effective treatment planning. Precise dental disease segmentation requires reliable tooth numbering, which may be prone to errors if performed manually. These steps can be automated using artificial intelligence, which may provide fast and accurate results. Among the AI methodologies, deep learning has recently shown excellent performance in dental image processing, allowing effective tooth segmentation and numbering. This paper proposes the Squeeze and Excitation Inception Block-based Encoder-Decoder (SE-IB-ED) network for teeth segmentation in panoramic X-ray images. It combines the InceptionV3 model for encoding with a custom decoder for feature integration and segmentation, using pointwise convolution and an attention mechanism. A dataset of 313 panoramic radiographs from private clinics was annotated using the Fédération Dentaire Internationale (FDI) system. PSPL and SAM augmented the annotation precision and effectiveness, with SAM automating teeth labeling and subsequently applying manual corrections. The proposed SE-IB-ED network was trained and tested using 80% training and 20% testing of the dataset, respectively. Data augmentation techniques were employed during training. It outperformed the state-of-the-art models with a very high F1-score of 92.65%, mIoU of 86.38%, and 92.84% in terms of accuracy, precision of 92.49%, and recall of 99.92% in the segmentation of teeth. According to the results obtained, the proposed method has great potential for the accurate segmentation of all teeth regions and backgrounds in panoramic X-ray images.

MSDFNet: multi-scale detail feature fusion encoder–decoder network for self-supervised monocular thermal image depth estimation

Abstract Currently available thermal image depth estimation methods are difficult to efficiently extract fine multi-scale feature information from thermal images and suffer from the problem of blurring details at the edges of the estimated depth map. To address these challenges, this paper proposes MSDFNet, a multi-scale detail feature fusion encoder–decoder network, for self-supervised monocular thermal image depth estimation. The model is based on a channel expansion hourglass residual lightweight feature encoder, which can capture rich and fine-grained multi-scale feature information with low computational effort. MSDFNet utilizes a detail feature weight evaluation decoder to fuse cross-scale features and reevaluate the importance of each feature, thereby emphasizing critical edge information at multiple scales. Additionally, MSDFNet incorporates a depth consistency loss function, which provides self-supervised signals for the detailed features of thermal images and improves the optimization of network performance. The method is applied to the ViViD++ and MS2 datasets and achieves state-of-the-art depth estimation performance compared to existing state-of-the-art algorithms. In the Indoor Dark scenario of the ViViD++ dataset, the Abs Rel, Sq Rel, RMSE, and RMSE log error metric values of MSDFNet are reduced by 6.71%, 11.92%, 9.09%, and 5.73%, respectively, while the accuracy metric values δ < 1.25 i , i = 1 , 2 , 3 were improved by 4.18%, 1.13%, and 0.2%, respectively. In addition, MSDFNet proves its excellent generalization ability on the MS2 dataset. The Abs Rel and RMSE error values in the night scene are reduced by 45.6% and 30.09%, respectively, and the accuracy δ < 1.25 i , i = 1 , 3 is improved by 20.95% and 1.33%, respectively. The Abs Rel and RMSE values in the rainy day scenario are reduced by 1.33% and 1.21%, respectively, and the accuracy δ < 1.25 i , i = 1 , 3 is improved by 0.24% and 0.83%, respectively.

Estimation of Beach Profile Response on Coastal Hydrodynamics Using LSTM-Based Encoder–Decoder Network

Beach profiles are constantly changing due to external ocean forces. Estimating these changes is crucial to understanding and addressing coastal erosion issues, such as shoreline advance and retreat. To estimate beach profile changes, obtaining long-term, high-resolution spatiotemporal beach profile data is essential. However, due to the limited availability of beach profile survey data both on land and underwater along the coast, generating continuous, high-resolution spatiotemporal beach profile data over extended periods is a critical technological challenge. Therefore, we herein developed a long short-term memory-based encoder–decoder network for effective spatiotemporal representation learning to estimate beach profile responses on temporal scales from weeks to months from coastal hydrodynamics. The proposed approach was applied to 12 transects from seven beaches located in three different littoral systems on the east coast of the Korean Peninsula, where coastal erosion problems are severe. The performance of the proposed method demonstrated improved results compared with a recent study that performed the same beach profile estimation task, with an average root mean square error of 0.50 m. Moreover, most of the results exhibited a reasonably accurate morphological shape of the estimated beach profile. However, instances where the results exceed the average error are attributed to extreme beach morphological changes caused by storm waves such as typhoons.

Latent Semantic and Disentangled Attention.

Sequential learning using transformer has achieved state-of-the-art performance in natural language tasks and many others. The key to this success is the multi-head self attention which encodes and gathers the features from individual tokens of an input sequence. The mapping or decoding is performed to produce an output sequence via cross attention. There are threefold weaknesses by using such an attention framework. First, since the attention would mix up the features of different tokens in input and output sequences, it is likely that redundant information exists in sequence data representation. Second, the patterns of attention weights among different heads tend to be similar. The model capacity is bounded. Third, the robustness in an encoder-decoder network against the model uncertainty is disregarded. To handle these weaknesses, this paper presents a Bayesian semantic and disentangled mask attention to learn latent disentanglement in multi-head attention where the redundant features in transformer are compensated with the latent topic information. The attention weights are filtered by a mask which is optimized through semantic clustering. This attention mechanism is implemented according to Bayesian learning for clustered disentanglement. The experiments on machine translation and speech recognition show the merit of Bayesian clustered disentanglement for mask attention.

Patient activity recognition for robust performance under varied illumination and cluttered background in indoor environment

Human Activity Recognition (HAR) holds significant importance in health and human-machine interaction. However, recognizing actions from 2D information faces challenges like occlusion, illumination variation, cluttered backgrounds, and view invariance. These hurdles are particularly pronounced in indoor patient monitoring settings due to fluctuating lighting conditions and cluttered backgrounds, which compromise the accuracy of activity recognition systems. A new architecture named illuminationRevive has been proposed to tackle this issue. Which utilizes an encoder-decoder convolutional neural network and image post-processing blocks to enhance the image's visual appearance. A new dataset comprising seven indoor physical activities has been proposed, created with contributions from thirty individuals aged 20 to 45. A hybrid fusion architecture is proposed to classify activities, integrating motion sequence information and body joint features. The proposed classification model incorporates generated Skeleton Motion History Images (SMHIs), collected human joint motion features from video frames, and novel kinematic and geometric features within window frames as inputs. The model can extract spatial and temporal feature vectors by integrating ResNet50-ViT (Residual Network-50 layers, Vision Transformer) and CNN-BiLSTM (Convolutional Neural Network-Bidirectional Long Short-Term Memory) layers. The suggested classification model was evaluated alongside state-of-the-art models using the LNMIIT-SMD (The LNM Institute of Information Technology-Skelton Motion Dataset) and established NTU-RGBD (Nanyang Technological University's Red Blue Green and Depth information) dataset. The evaluation aimed to assess the effectiveness of the proposed classification model architecture. Results demonstrate the superiority of the proposed model, achieving impressive accuracies of 98.21% on real-time data, 98.45% on the proposed dataset, and 97.12% on the NTU-RGBD dataset.

Root segmentation of horticultural plants in X-Ray CT images by integrating 2D instance segmentation with 3D point cloud clustering

X-ray computed tomography (CT) is a powerful tool for in situ plant root system architecture (RSA) characterization. Accurate root segmentation from CT images is integral to studying RSA. Research studies on segmenting roots from CT images have been mainly limited to image processing-based approaches which may require parameter tuning and often lack common segmentation metrics, e.g., Dice and IoU. A recent deep learning approach utilizes a volumetric encoder-decoder network to achieve a high Dice score and IoU. However, training a volumetric model is dependent on the availability of fully annotated scans of the growing medium column, obtaining which can be time-consuming, tedious, and resource intensive. In this study, an efficient method using deep learning-based instance segmentation in conjunction with density-based spatial clustering of applications with noise (DBSCAN)-based filtering was developed and evaluated for two horticultural plant species. A pretrained Mask R-CNN model was fine-tuned on images selected along different axes of the three-dimensional scans to identify the best view selection strategy for volumetric root segmentation. DBSCAN was used to filter noise from the volumetric segmentation with an automated parameter tuning technique. The proposed method was evaluated on scans of poinsettias and onions and achieved best average scores of 0.831, 0.839, 0.834, and 0.718 for Precision, Recall, Dice, and IoU, respectively. Further experiments showed reducing the training data to 1 % did not significantly impact the segmentation accuracy. Therefore, the proposed method has promising potential to facilitate RSA analysis with its high utility.

A bi‐stream transformer for single‐image dehazing

AbstractDeep‐learning methods, such as encoder–decoder networks, have achieved impressive results in image dehazing. However, these methods often rely only on synthesized data for training that limits their generalizability to hazy, real‐world images. To leverage prior knowledge of haze properties, we propose a bi‐encoder structure that integrates a prior‐based encoder into a traditional encoder–decoder network. The features from both encoders were fused using a feature enhancement module. We adopted transformer blocks instead of convolutions to model local feature associations. Experimental results demonstrate that our method surpasses state‐of‐the‐art methods for synthesized and actual hazy scenes. Therefore, we believe that our method will be a useful supplement to the collection of current artificial intelligence models and will benefit engineering applications in computer vision.