Encoder Architecture Research Articles

Forecasting Stock Market Indices Using Integration of Encoder, Decoder, and Attention Mechanism.

Accurate forecasting of stock market indices is crucial for investors, financial analysts, and policymakers. The integration of encoder and decoder architectures, coupled with an attention mechanism, has emerged as a powerful approach to enhance prediction accuracy. This paper presents a novel framework that leverages these components to capture complex temporal dependencies and patterns within stock price data. The encoder effectively transforms an input sequence into a dense representation, which the decoder then uses to reconstruct future values. The attention mechanism provides an additional layer of sophistication, allowing the model to selectively focus on relevant parts of the input sequence for making predictions. Furthermore, Bayesian optimization is employed to fine-tune hyperparameters, further improving forecast precision. Our results demonstrate a significant improvement in forecast precision over traditional recurrent neural networks. This indicates the potential of our integrated approach to effectively handle the complex patterns and dependencies in stock price data.

Exploring the Use of Deep Learning Models for Image Compression in Embedded Systems: Encoder and Decoder Architectures

With the growing demand for efficient image processing in embedded systems, the exploration of deep learning-based image compression methods has emerged as a promising avenue. Traditional image compression techniques, such as JPEG and PNG, face challenges in achieving optimal performance for constrained environments due to their reliance on handcrafted algorithms and limited adaptability. This study investigates the use of deep learning models for image compression tailored to embed systems, focusing on encoder and decoder architectures. By leveraging convolutional neural networks (CNNs) and variational auto encoders (VAEs), we design lightweight models capable of achieving high compression ratios while maintaining visual fidelity. The research emphasizes computational efficiency, ensuring compatibility with the resource constraints of embedded hardware. Key contributions include the development of streamlined architectures optimized for low memory and power usage, along with a comprehensive evaluation of compression quality, reconstruction accuracy, and real-time performance. The results demonstrate that deep learning-based approaches can outperform traditional methods in terms of adaptability and efficiency, paving the way for their integration into next-generation embedded systems.

Bit-rate aware effective inter-layer motion prediction using multi-loop encoding structure

Recently, there has been a notable increase in the use of video content on the internet, leading for the creation of improved codecs like versatile-video-coding (VVC) and high-efficiency video-coding (HEVC). It is important to note that these video coding techniques continue to demonstrate quality degradation and the presence of noise throughout the decoded frames. A number of deep-learning (DL) algorithm-based network structures have been developed by experts to tackle this problem; nevertheless, because many of these solutions use in-loop filtration, extra bits must be sent among the encoding and decoding layers. Moreover, because they used fewer reference frames, they were unable to extract significant features by taking advantage from the temporal connection between frames. Hence, this paper introduces inter-layer motion prediction aware multi-loop video coding (ILMPA-MLVC) techniques. The ILMPA-MLVC first designs an multi-loop adaptive encoder (MLAE) architecture to enhance inter-layer motion prediction and optimization process; second, this work designs multi-loop probabilistic-bitrate aware compression (MLPBAC) model to attain improved bitrate efficiency with minimal overhead; the training of ILMPA-MLVC is done through novel distortion loss function using UVG dataset; the result shows the proposed ILMPA-MLVC attain improved peak-singal-to-noise-ratio (PSNR) and structural similarity (SSIM) performance in comparison with existing video coding techniques.

Hand gestures classification of sEMG signals based on BiLSTM-metaheuristic optimization and hybrid U-Net-MobileNetV2 encoder architecture

Surface electromyography (sEMG) data has been extensively utilized in deep learning algorithms for hand movement classification. This paper aims to introduce a novel method for hand gesture classification using sEMG data, addressing accuracy challenges seen in previous studies. We propose a U-Net architecture incorporating a MobileNetV2 encoder, enhanced by a novel Bidirectional Long Short-Term Memory (BiLSTM) and metaheuristic optimization for spatial feature extraction in hand gesture and motion recognition. Bayesian optimization is employed as the metaheuristic approach to optimize the BiLSTM model’s architecture. To address the non-stationarity of sEMG signals, we employ a windowing strategy for signal augmentation within deep learning architectures. The MobileNetV2 encoder and U-Net architecture extract relevant features from sEMG spectrogram images. Edge computing integration is leveraged to further enhance innovation by enabling real-time processing and decision-making closer to the data source. Six standard databases were utilized, achieving an average accuracy of 90.23% with our proposed model, showcasing a 3–4% average accuracy improvement and a 10% variance reduction. Notably, Mendeley Data, BioPatRec DB3, and BioPatRec DB1 surpassed advanced models in their respective domains with classification accuracies of 88.71%, 90.2%, and 88.6%, respectively. Experimental results underscore the significant enhancement in generalizability and gesture recognition robustness. This approach offers a fresh perspective on prosthetic management and human-machine interaction, emphasizing its efficacy in improving accuracy and reducing variance for enhanced prosthetic control and interaction with machines through edge computing integration.

AFENet: An Attention-Focused Feature Enhancement Network for the Efficient Semantic Segmentation of Remote Sensing Images

The semantic segmentation of high-resolution remote sensing images (HRRSIs) faces persistent challenges in handling complex architectural structures and shadow occlusions, limiting the effectiveness of existing deep learning approaches. To address these limitations, we propose an attention-focused feature enhancement network (AFENet) with a novel encoder–decoder architecture. The encoder architecture combines ResNet50 with a parallel multistage feature enhancement group (PMFEG), enabling robust feature extraction through optimized channel reduction, scale expansion, and channel reassignment operations. Building upon this foundation, we develop a global multi-scale attention mechanism (GMAM) in the decoder that effectively synthesizes spatial information across multiple scales by learning comprehensive global–local relationships. The architecture is further enhanced by an efficient feature-weighted fusion module (FWFM) that systematically integrates remote spatial features with local semantic information to improve segmentation accuracy. Experimental results across diverse scenarios demonstrate that AFENet achieves superior performance in building structure detection, exhibiting enhanced segmentation connectivity and completeness compared to state-of-the-art methods.

Exploratory analysis of Type B Aortic Dissection (TBAD) segmentation in 2D CTA images using various kernels

Type-B Aortic Dissection is a rare but fatal cardiovascular disease characterized by a tear in the inner layer of the aorta, affecting 3.5 per 100,000 individuals annually. In this work, we explore the feasibility of leveraging two-dimensional Convolutional Neural Network (CNN) models to perform accurate slice-by-slice segmentation of true lumen, false lumen and false lumen thrombus in Computed Tomography Angiography images. The study performed an exploratory analysis of three 2D U-Net models: the baseline 2D U-Net, a variant of U-Net with atrous convolutions, and a U-Net with a custom layer featuring a position-oriented, partially shared weighting scheme kernel. These models were trained and benchmarked against a state-of-the-art baseline 3D U-Net model. Overall, our U-Net with the VGG19 encoder architecture achieved the best performance score among all other models, with a mean Dice score of 80.48% and an IoU score of 72.93%. The segmentation results were also compared with the Segment Anything Model (SAM) and the UniverSeg models. Our findings indicate that our 2D U-Net models excel in false lumen and true lumen segmentation accuracy while achieving lower false lumen thrombus segmentation accuracy compared to the state-of-the-art 3D U-Net model. The study findings highlight the complexities involved in developing segmentation models, especially for cardiovascular medical images, and emphasize the importance of developing lightweight models for real-time decision-making to improve overall patient care.

PE-based high throughput and low power polar encoder for 5G-NR PBCH channel

In this paper, we propose a PE-based parallelism configurable polar encoder hardware architecture for emerging high-speed 5G communication system. This encoder architecture is applied to the 5G-NR PBCH channel polar encoder, implemented with functional modules such as CRC generation and interleaving, channel encoding, and rate matching as specified in the 3GPP protocol. Next, based on the united power format (UPF) low-power management technology, the PBCH polar encoder architecture is divided into different power domains based on their operating characteristics to reduce the power consumption. Experimental results show that the proposed polar encoder achieves throughput up to 30 Gbps. Based on TSMC 40 nm CMOS technology, by applying the proposed low-power methodology, the power consumption of the PBCH polar encoder at parallelisms of 8, 16, and 32 are 1.236 mW, 1.170 mW, and 1.084 mW, achieving power reductions of 24%, 29%, and 35% when comparing to non-low-power design, respectively.

Investigating the impact of encoder architectures and batch size on depth estimation through semantic consistency

Investigating the impact of encoder architectures and batch size on depth estimation through semantic consistency

An efficient XOR-free implementation of polar encoder for reconfigurable hardware

— This paper presents a novel approach to implementing an XOR-Free architecture of the non-systematic polar encoder (NSPE) for 5G radio. The optimization of XOR logic for hardware (HW) implementation is essential to reduce delay and power consumption. The proposed architecture for NSPE replaces XOR operations with combinational logical patterns, and some redundant patterns are removed with the help of bit manipulation to make it more efficient. The design infers multiplexers (2:1 or 4:1) and inverters as its functional units, making the design adequate and effective in alleviating HW complexity. The XOR-Free encoder performs the same functionality as the XOR-based conventional encoder. We have written a MATLAB script that generates Verilog hardware description language (HDL) code for fully or partially parallel polar encoders tailored to specific code lengths (N) and degrees of parallelism (M). A comparative analysis of various fully and partially parallel encoders with the XOR-Free algorithm is presented. The implementation results show that the proposed architectures are more efficient in terms of HW cost, power consumption, throughput, and latency.

Generative haplotype prediction outperforms statistical methods for small variant detection in next-generation sequencing data.

Detection of germline variants in next-generation sequencing data is an essential component of modern genomics analysis. Variant detection tools typically rely on statistical algorithms such as de Bruijn graphs or Hidden Markov models, and are often coupled with heuristic techniques and thresholds to maximize accuracy. Despite significant progress in recent years, current methods still generate thousands of false-positive detections in a typical human whole genome, creating a significant manual review burden. We introduce a new approach that replaces the handcrafted statistical techniques of previous methods with a single deep generative model. Using a standard transformer-based encoder and double-decoder architecture, our model learns to construct diploid germline haplotypes in a generative fashion identical to modern large language models. We train our model on 37 whole genome sequences from Genome-in-a-Bottle samples, and demonstrate that our method learns to produce accurate haplotypes with correct phase and genotype for all classes of small variants. We compare our method, called Jenever, to FreeBayes, GATK HaplotypeCaller, Clair3, and DeepVariant, and demonstrate that our method has superior overall accuracy compared to other methods. At F1-maximizing quality thresholds, our model delivers the highest sensitivity, precision, and the fewest genotyping errors for insertion and deletion variants. For single nucleotide variants, our model demonstrates the highest sensitivity but at somewhat lower precision, and achieves the highest overall F1 score among all callers we tested. Jenever is implemented as a python-based command line tool. Source code is available at https://github.com/ARUP-NGS/jenever/.

Sea–Land Segmentation of Remote-Sensing Images with Prompt Mask-Attention

Remote-sensing technology has gradually become one of the most important ways to extract sea–land boundaries due to its large scale, high efficiency, and low cost. However, sea–land segmentation (SLS) is still a challenging problem because of data diversity and inconsistency, “different objects with the same spectrum” or “the same object with different spectra”, and noise and interference problems, etc. In this paper, a new sea–land segmentation method (PMFormer) for remote-sensing images is proposed. The contributions are mainly two points. First, based on Mask2Former architecture, we introduce the prompt mask by normalized difference water index (NDWI) of the target image and prompt encoder architecture. The prompt mask provides more reasonable constraints for attention so that the segmentation errors are alleviated in small region boundaries and small branches, which are caused by insufficiency of prior information by large data diversity or inconsistency. Second, for the large intra-class difference problem in the foreground–background segmentation in sea–land scenes, we use deep clustering to simplify the query vectors and make them more suitable for binary segmentation. Then, traditional NDWI and eight other deep-learning methods are thoroughly compared with the proposed PMFormer on three open sea–land datasets. The efficiency of the proposed method is confirmed, after the quantitative analysis, qualitative analysis, time consumption, error distribution, etc. are presented by detailed contrast experiments.

Olfactory Diagnosis Model for Lung Health Evaluation Based on Pyramid Pooling and SHAP-Based Dual Encoders.

This study introduces a novel deep learning framework for lung health evaluation using exhaled gas. The framework synergistically integrates pyramid pooling and a dual-encoder network, leveraging SHapley Additive exPlanations (SHAP) derived feature importance to enhance its predictive capability. The framework is specifically designed to effectively distinguish between smokers, individuals with chronic obstructive pulmonary disease (COPD), and control subjects. The pyramid pooling structure aggregates multilevel global information by pooling features at four scales. SHAP assesses feature importance from the eight sensors. Two encoder architectures handle different feature sets based on their importance, optimizing performance. Besides, the model's robustness is enhanced using the sliding window technique and white noise augmentation on the original data. In 5-fold cross-validation, the model achieved an average accuracy of 96.40%, surpassing that of a single encoder pyramid pooling model by 10.77%. Further optimization of filters in the transformer convolutional layer and pooling size in the pyramid module increased the accuracy to 98.46%. This study offers an efficient tool for identifying the effects of smoking and COPD, as well as a novel approach to utilizing deep learning technology to address complex biomedical issues.

Improving generative adversarial network inversion via fine-tuning GAN encoders

Generative adversarial networks (GANs) can synthesize high-quality (HQ) images, and GAN inversion is a technique that discovers how to invert given images back to latent space. While existing methods perform on StyleGAN inversion, they have limited performance and are not generalized to different GANs. To address these issues, we proposed a self-supervised method to pre-train and fine-tune GAN encoders. First, we designed an adaptive block to fit different encoder architectures for inverting diverse GANs. Then we pre-train GAN encoders using synthesized images and emphasize local regions through cropping images. Finally, we fine-tune the pre-trained GAN encoder for inverting real images. Compared with state-of-the-art methods, our method achieved better results that reconstructed high-quality images on mainstream GANs. Our code and pre-trained models are available at: https://github.com/disanda/Deep-GAN-Encoders.

What Kind of Transformer Models to Use for the ICD-10 Codes Classification Task.

Coding according to the International Classification of Diseases (ICD)-10 and its clinical modifications (CM) is inherently complex and expensive. Natural Language Processing (NLP) assists by simplifying the analysis of unstructured data from electronic health records, thereby facilitating diagnosis coding. This study investigates the suitability of transformer models for ICD-10 classification, considering both encoder and encoder-decoder architectures. The analysis is performed on clinical discharge summaries from the Medical Information Mart for Intensive Care (MIMIC)-IV dataset, which contains an extensive collection of electronic health records. Pre-trained models such as BioBERT, ClinicalBERT, ClinicalLongformer, and ClinicalBigBird are adapted for the coding task, incorporating specific preprocessing techniques to enhance performance. The findings indicate that increasing context length improves accuracy, and that the difference in accuracy between encoder and encoder-decoder models is negligible.

Geometric-aware RGB-D representation learning for hand–object reconstruction

Reconstructing hand–object interaction from single images, crucial for interactive applications, is challenging due to the diversity of hand–object poses and shapes. Depth maps effectively complement RGB data in understanding these interactions geometrically within challenging scenes. However, most existing methods do not fully utilize the potential benefits of RGB-D information fused at different feature levels for reconstruction, limiting their capability to capture the geometric details of hand–object interaction. To address this, we propose an implicit geometric-aware RGB-D representation learning approach using adaptive bidirectional RGB-D feature fusion (ABF) and geometric Fourier feature encoding (GFFE). This method innovatively leverages the hierarchical and complementary high-level features of RGB-D information to enhance the neural implicit representations of hand–object interaction during the feature extraction and encoding stages of RGB-D data. Initially, a two-stream RGB-point cloud encoder, combining CNN and Transformer architectures, extracts appearance and geometric information from RGB and point cloud data. Subsequently, ABF fuses this information, generating dense RGB-D features by leveraging the complementary of appearance and geometric features along with the varying sensitivities of different layers. Finally, GFFE introduces frequency domain information to capture both high-frequency and low-frequency features of objects equally to precisely capture the geometric details of hand–object interaction. Experiments conducted on the real-world DexYCB and the synthetic ObMan benchmarks demonstrate that our approach significantly outperforms existing approaches.

Flexible and High-Efficiency LDPC Encoder Architecture for CCSDS Standard

Abstract The Consultative Committee for Space Data Systems (CCSDS) has adopted quasi-cyclic low-density parity-check (QC-LDPC) codes for use in near-Earth (C2) and deep space (AR4JA) communications. Existing encoder architectures for C2 codes, however, fall short in efficiency for high-throughput applications. This paper introduces a comprehensive approach combining algorithmic and architectural optimizations to enhance hardware usage efficiency (HUE) while offering flexibility. We propose an integrated inter-block and intra-block parallel (IIB-IBP) encoding algorithm that leverages the unique matrix structure to significantly enhance performance. Additionally, a matrix-specific command register pretreatment (MSCRP) technique is developed to effectively handle the special dimensions of the generator matrix. Furthermore, we detail an offline design process for the automated generation of the encoder core’s HDL description, facilitating fine-tuning of encoding parallelism, latency, FPGA resource utilization, and overall throughput. Hardware implementation on a Virtex XC5VLX110T FPGA demonstrates that our encoder reaches an impressive throughput of 10.6 Gb/s with only 2531 LUTs and 1040 FFs, achieving a HUE of 2.97 Mbps/logic unit. This performance marks a 70.6% increase in HUE when compared to state-of-the-art designs.

A duplex transform heterogeneous feature fusion network for road segmentation

Detecting roads in automatic driving environments poses a challenge due to issues such as boundary fuzziness, occlusion, and glare from light. We believe that two factors are instrumental in addressing these challenges and enhancing detection performance: global context dependency and effective feature representation that prioritizes important feature channels. To tackle these issues, we introduce DTRoadseg, a novel duplex Transformer-based heterogeneous feature fusion network designed for road segmentation. DTRoadseg leverages a duplex encoder architecture to extract heterogeneous features from both RGB images and point-cloud depth images. Subsequently, we introduce a multi-source Heterogeneous Feature Reinforcement Block (HFRB) for fusion of the encoded features, comprising a Heterogeneous Feature Fusion Module (HFFM) and a Reinforcement Fusion Module (RFM). The HFFM leverages the self-attention mechanisms of Transformers to achieve effective fusion through token interactions, while the RFM focuses on emphasizing informative features while downplaying less important ones, thereby reinforcing feature fusion. Finally, a Transformer decoder is utilized to produce the final semantic prediction. Furthermore, we employ a boundary loss function to optimize the segmentation structure area, reduce false detection areas, and improve model accuracy. Extensive experiments are carried out on the KITTI road dataset. The results demonstrate that, compared with state-of-the-art methods, DTRoadseg exhibits superior performance, achieving an average accuracy of 97.01%, a Recall of 96.35%, and runs at a speed of 0.09 s per picture.

Pest-ConFormer: A hybrid CNN-Transformer architecture for large-scale multi-class crop pest recognition

Crop pests are acknowledged as the major factors in reducing the yield and quality of agricultural production worldwide. It is an urgent necessity to recognize crop pests accurately to protect the crop in the early stage to reduce the loss for the agricultural economy. Due to the ecological characteristics of the crop pests and the complex background in fields, the crop pests show high inter-class similarity and significant intra-class variation in external morphology appearance, which makes current recognition methods suffer from low classification accuracy and poor generalization ability in complex natural environment recognition tasks. To tackle this problem, a hybrid convolutional neural network and transformer-based model, namely Pest-ConFormer, featured with multi-scale weakly supervised feature selection mechanisms is proposed, which has shown excellent multi-scale discriminative feature extraction in fine-grained visual classification (FGVC) tasks. This method employs a hybrid convolution-transformer encoder architecture pre-training in a self-supervised masked autoencoder manner as a backbone to learn pests’ highly discriminative features across various scales. Next, a dual-path feature aggregation structure with a top-down FPN-like feature pathway and a bottom-up PANet-like feature pathway based on attention mechanisms is designed to learn high-level global context information and low-level local detailed feature representation. Thirdly, a fine-grained classification module using weakly supervised learning is introduced to select the discriminative feature points in different pyramidal levels. Then, these feature points are fed into a graph convolutional network to accomplish classification. Several experiments are conducted on the large-scale multi-class IP102 benchmark dataset, and the proposed method achieves an accuracy of 77.81 % regarding crop pest recognition. The experimental results indicate that our approach outperforms other state-of-the-art methods by nearly 2 percent points, demonstrating that the proposed hybrid architecture with dual-path feature aggregation and fine-grained classification modules can be more effective in the crop pest recognition field than CNN-based methods and can be deployed in the practical natural environment. The source code will be available at https://github.com/mwfang/pestconformer.

Failure Mode Classification for Rolling Element Bearings Using Time-Domain Transformer-Based Encoder.

In this paper, we propose a Transformer-based encoder architecture integrated with an unsupervised denoising method to learn meaningful and sparse representations of vibration signals without the need for data transformation or pre-trained data. Existing Transformer models often require transformed data or extensive computational resources, limiting their practical adoption. We propose a simple yet competitive modification of the Transformer model, integrating a trainable noise reduction method specifically tailored for failure mode classification using vibration data directly in the time domain without converting them into other domains or images. Furthermore, we present the key architectural components and algorithms underlying our model, emphasizing interpretability and trustworthiness. Our model is trained and validated using two benchmark datasets: the IMS dataset (four failure modes) and the CWRU dataset (four and ten failure modes). Notably, our model performs competitively, especially when using an unbalanced test set and a lightweight architecture.

Enhancing Self-Supervised Learning through Explainable Artificial Intelligence Mechanisms: A Computational Analysis

Self-supervised learning continues to drive advancements in machine learning. However, the absence of unified computational processes for benchmarking and evaluation remains a challenge. This study conducts a comprehensive analysis of state-of-the-art self-supervised learning algorithms, emphasizing their underlying mechanisms and computational intricacies. Building upon this analysis, we introduce a unified model-agnostic computation (UMAC) process, tailored to complement modern self-supervised learning algorithms. UMAC serves as a model-agnostic and global explainable artificial intelligence (XAI) methodology that is capable of systematically integrating and enhancing state-of-the-art algorithms. Through UMAC, we identify key computational mechanisms and craft a unified framework for self-supervised learning evaluation. Leveraging UMAC, we integrate an XAI methodology to enhance transparency and interpretability. Our systematic approach yields a 17.12% increase in improvement in training time complexity and a 13.1% boost in improvement in testing time complexity. Notably, improvements are observed in augmentation, encoder architecture, and auxiliary components within the network classifier. These findings underscore the importance of structured computational processes in enhancing model efficiency and fortifying algorithmic transparency in self-supervised learning, paving the way for more interpretable and efficient AI models.