Encoder-decoder Architecture Research Articles

An attention enriched encoder–decoder architecture with CLSTM and RES unit for segmenting exudate in retinal images

An attention enriched encoder–decoder architecture with CLSTM and RES unit for segmenting exudate in retinal images

A Multicomponent Linear Frequency Modulation Signal-Separation Network for Multi-Moving-Target Imaging in the SAR-Ground-Moving-Target Indication System

Multi-moving-target imaging in a synthetic aperture radar (SAR) system poses a significant challenge owing to target defocusing and being contaminated by strong background clutter. Aiming at this problem, a new deep-convolutional-neural-network (CNN)-assisted method is proposed for multi-moving-target imaging in a SAR-GMTI system. The multi-moving-target signal can be modeled by a multicomponent LFM signal with additive perturbation. A fully convolutional network named MLFMSS-Net was designed based on an encoder–decoder architecture to extract the most-energetic LFM signal component from the multicomponent LFM signal in the time domain. Without prior knowledge of the target number, an iterative signal-separation framework based on the well-trained MLFMSS-Net is proposed to separate the multi-moving-target signal into multiple LFM signal components while eliminating the residual clutter. It works well, exhibiting high imaging robustness and low dependence on the system parameters, making it a suitable solution for practical imaging applications. Consequently, a well-focused multi-moving-target image can be obtained by parameter estimation and secondary azimuth compression for each separated LFM signal component. The simulations and experiments on both airborne and spaceborne SAR data showed that the proposed method is superior to traditional imaging methods in both imaging quality and efficiency.

Analysis of the combustion modes in a rocket-based combined cycle combustor using unsupervised machine learning methodology

Combustion mode analysis is essential to a rocket-based combined cycle (RBCC) combustor because it may experience multiple combustion modes during the operation. In this study, a method based on an autoencoder and a K-means algorithm was proposed for combustion mode analysis. Flame chemiluminescence images and schlieren images of three combustion modes observed in an RBCC combustor were used to evaluate this method. Two autoencoders that followed the same encoder–decoder architecture were developed separately to generate the latent space representations of flame chemiluminescence images and schlieren images. In the latent space, the centroids and boundaries of different combustion modes were determined using the K-means algorithm. Each autoencoder was trained using 750 images and tested using another 3000 images. The method achieved an accuracy up to 99% on both flame chemiluminescence images and schlieren images. The images generated by the decoder suggested that the autoencoder captured the important features (e.g., primary reaction zone and shock wave) of the reacting flow field. The autoencoder developed for flame chemiluminescence images also successfully detected the combustion mode transition during an ignition process, which suggested that it had the potential to monitor the combustion mode in a real time manner. However, the autoencoder failed on monitoring combustion mode transition when it came to the schlieren images because the optical access of the training data was not exactly the same. Therefore, it is essential to ensure that the optical accesses of different combustion modes are exactly the same when schlieren images are employed for combustion mode analysis.

SalDA: DeepConvNet Greets Attention for Visual Saliency Prediction

Predicting salient regions in images requires the capture of contextual information in the scene. Conventional saliency models typically use the encoder-decoder architecture and multi-scale feature fusion for modeling contextual features, which, however, possess huge computational cost and model parameters. In this paper, we address the saliency prediction task by capturing long-range dependencies based on the self-attention mechanism. Self-attention has been widely used in image recognition or other classification tasks, but is still rarely being considered in regression based saliency prediction task. Inspired by non-local block, we propose a new SALiency prediction network in which DeepConvNet is integrated with the Attention mechanism, namely SalDA. Considering each feature map may capture different salient regions, our spatial attention module firstly adaptively aggregates the feature at each position by a weighted sum of the features at all positions within each independent channel. Meanwhile, in order to capture interdependence between channels, we also introduce a channel attention module to integrate different features among different channels. We combine these two attention modules into a multi-attention module to further improve the saliency map prediction for the network. We show the effectiveness of SalDA on the largest saliency prediction dataset SALICON. Compared to other state-of-the-art methods in this area, we can yield comparable saliency prediction performance, but with substantially less model parameters and shorter inference time.

A segmentation network for farmland ridge based on encoder-decoder architecture in combined with strip pooling module and ASPP.

In order to effectively support wheat breeding, farmland ridge segmentation can be used to visualize the size and spacing of a wheat field. At the same time, accurate ridge information collecting can deliver useful data support for farmland management. However, in the farming ridge segmentation scenarios based on remote sensing photos, the commonly used semantic segmentation methods tend to overlook the ridge edges and ridge strip features, which impair the segmentation effect. In order to efficiently collect ridge information, this paper proposes a segmentation method based on encoder-decoder of network with strip pooling module and ASPP module. First, in order to extract context information for multi-scale features, ASPP module are integrated in the deepest feature map. Second, the remote dependence of the ridge features is improved in both horizontal and vertical directions by using the strip pooling module. The final segmentation map is generated by fusing the boundary features and semantic features using an encoder and decoder architecture. As a result, the accuracy of the proposed method in the validation set is 98.0% and mIoU is 94.6%. The results of the experiments demonstrate that the method suggested in this paper can precisely segment the ridge information, as well as its value in obtaining data on the distribution of farmland and its potential for practical application.

Word2Pix: Word to Pixel Cross-Attention Transformer in Visual Grounding.

Current one-stage methods for visual grounding encode the language query as one holistic sentence embedding before fusion with visual features for target localization. Such a formulation provides insufficient ability to model query at the word level, and therefore is prone to neglect words that may not be the most important ones for a sentence but are critical for the referred object. In this article, we propose Word2Pix: a one-stage visual grounding network based on the encoder-decoder transformer architecture that enables learning for textual to visual feature correspondence via word to pixel attention. Each word from the query sentence is given an equal opportunity when attending to visual pixels through multiple stacks of transformer decoder layers. In this way, the decoder can learn to model the language query and fuse language with the visual features for target prediction simultaneously. We conduct the experiments on RefCOCO, RefCOCO+, and RefCOCOg datasets, and the proposed Word2Pix outperforms the existing one-stage methods by a notable margin. The results obtained also show that Word2Pix surpasses the two-stage visual grounding models, while at the same time keeping the merits of the one-stage paradigm, namely, end-to-end training and fast inference speed. Code is available at https://github.com/azurerain7/Word2Pix.

Deep learning-based segmentation of breast masses using convolutional neural networks

Automatic breast tumor segmentation based on convolutional neural networks (CNNs) is significant for the diagnosis and monitoring of breast cancers. CNNs have become an important method for early diagnosis of breast cancer and, thus, can help decrease the mortality rate. In order to assist medical professionals in breast cancer investigation a computerized system based on two encoder-decoder architectures for breast tumor segmentation has been developed. Two pre-trained DeepLabV3+ and U-Net models are proposed. The encoder generates a high-dimensional feature vector while the decoder analyses the low-resolution feature vector provided by the encoder and generates a semantic segmentation mask. Semantic segmentation based on deep learning techniques can overcome the limitations of traditional algorithms. To assess the efficiency of breast ultrasound image segmentation, we compare the segmentation results provided by CNNs against the Local Graph Cut technique (a semi-automatic segmentation method) in the Image Segmenter application. The output segmentation results have been evaluated by using the Dice similarity coefficient that compares the ground truth images provided by the specialists against the predicted segmentation results provided by the CNNs and Local Graph Cut algorithm. The proposed approach is validated on 780 breast ultrasonographic images of the BUSI public database of which 437 are benign and 210 are malignant. The BUSI database provides classification (benign or malignant) labels for ground truth in binary mask images. The average Dice scores computed between the ground truth images against CNNs were as follows: 0.9360 (malignant) and 0.9325 (benign) for the DeepLabV3+ architecture and of 0.6251 (malignant) and 0.6252 (benign) for the U-Net, respectively. When the segmentation results provided by CNNs were compared with the Local Graph Cut segmented images, the Dice scores were 0.9377 (malignant) and 0.9204 (benign) for DeepLabV3+ architecture and 0.6115 (malignant) and 0.6119 (benign) for U-Net, respectively. The results show that the DeepLabV3+ has significantly better segmentation performance and outperforms the U-Net network.

High-resolution building extraction based on the edge-aware network CEEAU_Net

Spatial information such as building location and distribution plays an important role in urban dynamic monitoring and urban planning applications. In recent years, deep learning methods have developed rapidly and achieved state-of-the-art performance in building extraction from remote sensing images in a variety of scenarios. However, existing semantic segmentation models pay more attention to global semantic information, emphasize multi-scale feature fusion or set lighter acceptance domains to obtain more global features, and ignore low-level detail features such as edges. Therefore, a new end-to-end deep learning network CEEAU_Net based on encoder-decoder architecture is designed to add edge sensing module and edge feature extraction module to obtain edge feature information of buildings. The Luxian county area of Luzhou City, Sichuan province is selected for building dataset production, which is located in the Longmenshan seismic zone, with many earthquakes of magnitude three or above, and the scene is complex, so a more accurate building extraction method is needed. Comparison experiments are also conducted with several advanced models on two public datasets, WHU building dataset (WHU) and Massachusetts. Selection of multiple indicators for indicator evaluation of results. CEEAU_Net achieves the best results in the metrics of overall accuracy, F1-score, Intersection over Union (IoU) and Mean Intersection over Union (MIoU), which suggests that the method proposed in this paper can effectively improve the accuracy of building extraction.

TrashInst: real-time floating litter instance segmentation model

This paper introduces TrashInst, an innovative fully convolutional single-stage anchor-free real-time instance segmentation model designed specifically for the detection of floating litter in waterways. TrashInst features a streamlined Encoder-Decoder architecture and an efficient channel attention mask head, ensuring the preservation of critical feature maps while minimizing redundant computations. Even in scenarios with limited feature maps, our approach, complemented by sparse instance activation maps, excels in both detection and segmentation tasks. Our encoder effectively processes features derived from ResNet50 across multiple scales, generating comprehensive abstractions through the nested feature module. Subsequently, the Vortex Instance Activation Maps module (VIAM) aggregates these multi-scale nested features and integrates them with mask head outputs, yielding precise instance masks. Furthermore, we employ the focal Tversky objective function to balance the dataset. In terms of performance, our approach outperforms state-of-the-art real-time instance segmentation models, achieving a remarkable 34% accuracy improvement, all while maintaining swift execution at 44 frames per second (FPS) on a single NVIDIA GTX-3090 GPU. Rigorous validation underscores the delicate interplay between speed and precision, highlighting the model’s exceptional ability to handle objects of varying sizes in real-time settings. Significantly, TrashInst attains superior average precision (AP) results in detecting large to medium-sized objects, outperforming existing models by a notable 4∼22% in the large category and an impressive 1.1∼56% in the medium category within our dataset. For the benefit of the community data will be available at (https://github.com/nassim12/TrashInst).

Deep-learning framework for optimal selection of soil sampling sites

Soil sampling is one of the most fundamental agricultural processes. The selection of soil sampling locations within a field plays a crucial role in soil-health analysis. The current soil-sampling techniques produce random samples across a field, which are not representative of the field conditions. This work leverages the recent advancements of deep learning in image processing for finding optimal locations that present the important characteristics of a field. The data for training are collected at different fields in local farms with five features: aspect, flow accumulation, slope, NDVI (normalized difference vegetation index), and yield. The soil sampling dataset is challenging because of the unbalance between the number of pixels that represent soil-sampling spots and the number of pixels that represent the background area. In this work, we approach the problem with two methods, the first approach involves utilizing a state-of-the-art model with the convolutional neural network (CNN) backbone, while the second is to innovate a deep-learning design grounded in the concepts of transformer and self-attention. Our framework is constructed with an encoder–decoder architecture with the self-attention mechanism as the backbone. In the encoder, the self-attention mechanism is the key feature extractor, which produces feature maps. In the decoder, we introduce atrous convolution networks to concatenate, fuse the extracted features, and then export the optimal locations for soil sampling. Currently, the model has achieved impressive results on the testing dataset, with a mean accuracy of 99.52%, a mean Intersection over Union (IoU) of 57.35%, and a mean Dice Coefficient of 71.47%, while the performance metrics of the state-of-the-art CNN-based model are 66.08%, 3.85%, and 1.98%, respectively. This indicates that our proposed model outperforms the CNN-based method on the soil-sampling dataset. To the best of our knowledge, our work is the first to provide a soil-sampling dataset with multiple attributes and leverage deep learning techniques to enable the automatic selection of soil-sampling sites. This work lays a foundation for novel applications of data science and machine-learning technologies to solve other emerging agricultural problems.

InClust+: the deep generative framework with mask modules for multimodal data integration, imputation, and cross-modal generation

BackgroundWith the development of single-cell technology, many cell traits can be measured. Furthermore, the multi-omics profiling technology could jointly measure two or more traits in a single cell simultaneously. In order to process the various data accumulated rapidly, computational methods for multimodal data integration are needed.ResultsHere, we present inClust+, a deep generative framework for the multi-omics. It’s built on previous inClust that is specific for transcriptome data, and augmented with two mask modules designed for multimodal data processing: an input-mask module in front of the encoder and an output-mask module behind the decoder. InClust+ was first used to integrate scRNA-seq and MERFISH data from similar cell populations, and to impute MERFISH data based on scRNA-seq data. Then, inClust+ was shown to have the capability to integrate the multimodal data (e.g. tri-modal data with gene expression, chromatin accessibility and protein abundance) with batch effect. Finally, inClust+ was used to integrate an unlabeled monomodal scRNA-seq dataset and two labeled multimodal CITE-seq datasets, transfer labels from CITE-seq datasets to scRNA-seq dataset, and generate the missing modality of protein abundance in monomodal scRNA-seq data. In the above examples, the performance of inClust+ is better than or comparable to the most recent tools in the corresponding task.ConclusionsThe inClust+ is a suitable framework for handling multimodal data. Meanwhile, the successful implementation of mask in inClust+ means that it can be applied to other deep learning methods with similar encoder-decoder architecture to broaden the application scope of these models.

WGformer: A Weibull-Gaussian Informer based model for wind speed prediction

Accurate wind speed forecasting can improve energy management efficiency and promote the use of renewable energy. However, the inherent nonlinearity and fluctuation of wind speed make prediction challenging. To address these issues, we design an efficient Informer-based model, with improved calculation speed, forecasting accuracy and generalization ability. The proposed model in this paper reasonably integrates the Weibull-Gaussian transform, Informer and kernel mean square error loss and addresses the combination of various components. The Weibull-Gaussian transform is used as the data preprocessing module, which can remove non-Gaussian characteristics from the original data, and thus achieve noise reduction. The Informer is used as the main predictor, which can efficiently output accurate forecasting results based on an encoder-decoder architecture and self-attention mechanism. The kernel mean square error loss function, which shows strong robustness to outliers, is used to evaluate the nonlinearity of errors in reproducing kernel Hilbert space. To evaluate the performance of the proposed model, it is compared with several widely used models and state-of-the-art models. The experimental results indicate that the proposed model weakens the effect of outliers, yields high forecasting accuracy with mean square error = 0.35, and outperforms the baselines up to 8.5% on three datasets.

Enhanced encoder–decoder architecture for visual perception multitasking of autonomous driving

Visual perception plays a vital role in autonomous driving systems, demanding high accuracy and real-time inference speed to ensure safety. In this paper, we propose a multi-task framework that simultaneously performs object detection, drivable area segmentation, and lane line identification, addressing the requirements of accurate and efficient visual perception. Our approach utilizes a shared-encoder architecture with three separate decoders, targeting each specific task. We investigate three configurations for the shared encoder: a Convolutional Neural Network (CNN), a Polyp Vision Transformer (PVT), and a hybrid CNN+PVT model. Through extensive experimentation and comparative analysis on the challenging BD100K dataset, we evaluate the performance of these shared-encoder models and provide valuable insights into their strengths and weaknesses. Our research contributes to the advancement of multi-task visual perception for autonomous driving systems by achieving competitive results in terms of accuracy and efficiency. The source code is publicly available on GitHub to facilitate further research in this domain.

Arbitrary Timestep Video Frame Interpolation with Time-Dependent Decoding

Given an observed low frame rate video, video frame interpolation (VFI) aims to generate a high frame rate video, which has smooth video frames with higher frames per second (FPS). Most existing VFI methods often focus on generating one frame at a specific timestep, e.g., 0.5, between every two frames, thus lacking the flexibility to increase the video’s FPS by an arbitrary scale, e.g., 3. To better address this issue, in this paper, we propose an arbitrary timestep video frame interpolation (ATVFI) network with time-dependent decoding. Generally, the proposed ATVFI is an encoder–decoder architecture, where the interpolation timestep is an extra input added to the decoder network; this enables ATVFI to interpolate frames at arbitrary timesteps between input frames and to increase the video’s FPS at any given scale. Moreover, we propose a data augmentation method, i.e., multi-width window sampling, where video frames can be split into training samples with multiple window widths, to better leverage training frames for arbitrary timestep interpolation. Extensive experiments were conducted to demonstrate the superiority of our model over existing baseline models on several testing datasets. Specifically, our model trained on the GoPro training set achieved 32.50 on the PSNR metric on the commonly used Vimeo90k testing set.

Attention-guided generator with dual discriminator GAN for real-time video anomaly detection

Detecting anomalies in videos presents a significant challenge in the field of video surveillance. The primary goal is identifying and detecting uncommon actions or events within a video sequence. The difficulty arises from the limited availability of video frames depicting anomalies and the ambiguous definition of anomaly. Based on extensive applications of Generative Adversarial Networks (GANs), which consist of a generator and a discriminator network, we propose an Attention-guided Generator with Dual Discriminator GAN (A2D-GAN) for real-time video anomaly detection (VAD). The generator network uses an encoder–decoder architecture with a multi-stage self-attention added to the encoder and multi-stage channel attention added to the decoder. The framework uses adversarial learning from noise and video frame reconstruction to enhance the generalization of the generator network. Also, of the dual discriminator in A2D-GAN, one discriminates between the reconstructed video frame and the real video frame, while the other discriminates between the reconstructed noise and the real noise. Exhaustive experiments and ablation studies on four benchmark video anomaly datasets, namely UCSD Peds, CUHK Avenue, ShanghaiTech, and Subway, demonstrate the effectiveness of the proposed A2D-GAN compared to other state-of-the-art methods. The proposed A2D-GAN model is robust and can detect anomalies in videos in real-time. The source code to replicate the results of the proposed A2D-GAN model is available at https://github.com/Rituraj-ksi/A2D-GAN.

DSLSTM: a deep convolutional encoder–decoder architecture for abnormality detection in video surveillance

DSLSTM: a deep convolutional encoder–decoder architecture for abnormality detection in video surveillance

Outlier item detection in bundle recommendation via the attention mechanism

Bundle recommendation offers users more holistic insights by recommending multiple compatible items at once. However, the intricate correlations between items, varied user preferences, and the pronounced data sparsity in combinations present significant challenges for bundle recommendation algorithms. Furthermore, current bundle recommendation methods fail to identify mismatched items within a given set, a process termed as “outlier item detection”. These outlier items are those with the weakest correlations within a bundle. Identifying them can aid users in refining their item combinations. While the correlation among items can predict the detection of such outliers, the adaptability of combinations might not be adequately responsive to shifts in individual items during the learning phase. This limitation can hinder the algorithm’s performance. To tackle these challenges, we introduce an encoder–decoder architecture tailored for outlier item detection. The encoder learns potential item correlations through a self-attention mechanism. Concurrently, the decoder garners efficient inference frameworks by directly assessing item anomalies. We have validated the efficacy and efficiency of our proposed algorithm using real-world datasets.

TSCA-Net: Transformer based spatial-channel attention segmentation network for medical images

Deep learning architectures based on convolutional neural network (CNN) and Transformer have achieved great success in medical image segmentation. Models based on the encoder–decoder framework like U-Net have been successfully employed in many realistic scenarios. However, due to the low contrast between object and background, various shapes and scales of objects, and complex background in medical images, it is difficult to locate targets and obtain better segmentation performance by extracting effective information from images. In this paper, an encoder–decoder architecture based on spatial and channel attention modules built by Transformer is proposed for medical image segmentation. Concretely, spatial and channel attention modules based on Transformer are utilized to extract spatial and channel global complementary information at different layers in U-shape network, which is beneficial to learn the detail features in different scales. To fuse better spatial and channel information from Transformer features, a spatial and channel feature fusion block is designed for the decoder. The proposed network inherits the advantages of both CNN and Transformer with the local feature representation and long-range dependency for medical images. Qualitative and quantitative experiments demonstrate that the proposed method outperforms against eight state-of-the-art segmentation methods on five publicly medical image datasets including different modalities, such as 80.23% and 93.56% Dice value, 67.13% and 88.94% Intersection over Union (IoU) value on the Multi-organ Nucleus Segmentation (MoNuSeg) and Combined Healthy Abdominal Organ Segmentation with Computed Tomography scans (CHAOS-CT) datasets.

Generating diverse clothed 3D human animations via a generative model

Data-driven garment animation is a current topic of interest in the computer graphics industry. Existing approaches generally establish the mapping between a single human pose or a temporal pose sequence, and garment deformation, but it is difficult to quickly generate diverse clothed human animations. We address this problem with a method to automatically synthesize dressed human animations with temporal consistency from a specified human motion label. At the heart of our method is a two-stage strategy. Specifically, we first learn a latent space encoding the sequence-level distribution of human motions utilizing a transformer-based conditional variational autoencoder (Transformer-CVAE). Then a garment simulator synthesizes dynamic garment shapes using a transformer encoder–decoder architecture. Since the learned latent space comes from varied human motions, our method can generate a variety of styles of motions given a specific motion label. By means of a novel beginning of sequence (BOS) learning strategy and a self-supervised refinement procedure, our garment simulator is capable of efficiently synthesizing garment deformation sequences corresponding to the generated human motions while maintaining temporal and spatial consistency. We verify our ideas experimentally. This is the first generative model that directly dresses human animation.

Lossless Text Compression Using Recurrent Neural Networks

Lossless Data compression is the process of reducing the size or the number of bits required to represent data, and Arithmetic coding is one of the popular lossless text compression techniques. This project focuses on lossless data compression using an innovative approach that combines arithmetic coding and Recurrent Neural Networks (RNN) with the help of encoder-decoder architecture. The proposed method aims to improve compression ratios compared to standard techniques. By leveraging the power of RNNs and arithmetic coding, encoder-decoder architecture effectively captures patterns and dependencies in the input data, enabling more efficient compression. The proposed approach achieved a remarkable improvement, achieving an impressive compression ratio that is, on average, 3.5 times smaller than that achieved by traditional methods. These results showcase the potential of developing improved general-purpose compressors based on neural networks and hybrid modeling. This demonstrates the potential of the proposed method for significantly enhancing lossless data compression performance.