Encoder Network Research Articles

Quantitative Stock Selection Model Using Graph Learning and a Spatial–Temporal Encoder

In the rapidly evolving domain of finance, quantitative stock selection strategies have gained prominence, driven by the pursuit of maximizing returns while mitigating risks through sophisticated data analysis and algorithmic models. Yet, prevailing models frequently neglect the fluid dynamics of asset relationships and market shifts, a gap that undermines their predictive and risk management efficacy. This oversight renders them vulnerable to market volatility, adversely affecting investment decision quality and return consistency. Addressing this critical gap, our study proposes the Graph Learning Spatial–Temporal Encoder Network (GL-STN), a pioneering model that seamlessly integrates graph theory and spatial–temporal encoding to navigate the intricacies and variabilities of financial markets. By harnessing the inherent structural knowledge of stock markets, the GL-STN model adeptly captures the nonlinear interactions and temporal shifts among assets. Our innovative approach amalgamates graph convolutional layers, attention mechanisms, and long short-term memory (LSTM) networks, offering a comprehensive analysis of spatial–temporal data features. This integration not only deciphers complex stock market interdependencies but also accentuates crucial market insights, enabling the model to forecast market trends with heightened precision. Rigorous evaluations across diverse market boards—Main Board, SME Board, STAR Market, and ChiNext—underscore the GL-STN model’s exceptional ability to withstand market turbulence and enhance profitability, affirming its substantial utility in quantitative stock selection.

Deep incomplete multi-view clustering via attention-based direct contrastive learning

Incomplete Multi-View Clustering (IMVC), enhanced by contrastive learning, stands out in unsupervised learning for its notable performance. However, it faces challenges: over-reliance on additional projection heads to avoid dimensionality collapse, leading to redundant parameters, and the risk of encoder-derived features, which merge view-specific information, misleading the learning of common semantics due to simultaneous learning and reconstruction in the same feature space. To address these issues, we propose a novel framework for incomplete multi-view contrastive clustering. This framework employs an encoder network with a self-attention mechanism, allowing both reconstruction loss and contrastive loss to act on the learned feature vectors and their sub-vectors, respectively. This approach effectively mitigates the impact of extraneous private information. By leveraging sub-vectors for consistency learning, our model directly refines the latent feature subspace, thus circumventing dimensionality collapse without the dependence on projection heads. Additionally, our method incorporates a cross-view prediction mechanism to recuperate missing information in incomplete datasets. Comprehensive experiments on public datasets demonstrate that our method achieves state-of-the-art clustering performance.

Deep learning-based channel estimation with application to 5G and beyond networks

Channel state information (CSI) feedback estimation for a downlink medium in a massive multiple input multiple output (MIMO) system is an essential and critical task to improve channel capacity and performance yield, especially in a frequency division duplex (FDD) multiplexing system. However, spectral efficiency degradation is a massive issue due to high channel feedback overhead. This work proposes a deep learning-based channel estimation (DLCE) model to improve channel reconstruction efficiency and channel overhead reduction accuracy. The proposed deep learning (DL) mechanism consists of encoder and decoder network where encoder network is utilized to compress CSI matrices whereas decoder network is used to decompress obtained CSI matrices. Here, inverse discrete Fourier transform (IDFT) method is utilized to convert CSI matrices of frequency domain into CSI matrices of delay domain. Simulation results are evaluated between uplink and downlink medium in the massive MIMO system considering a co-operation in science and technology (COST) 2,100 model. Here, a significant improvement in correlation and normalized mean square error (NMSE) results is observed. The proposed DLCE model shows superior performance against varied channel estimation techniques in terms of NMSE and correlation efficiency.

Improved wafer map defect pattern classification using automatic data augmentation based lightweight encoder network in contrastive learning

Improved wafer map defect pattern classification using automatic data augmentation based lightweight encoder network in contrastive learning

NCME-Net: Nuclear cataract mask encoder network for intelligent grading using self-supervised learning from anterior segment photographs

Cataracts are a leading cause of blindness worldwide, making accurate diagnosis and effective surgical planning critical. However, grading the severity of the lens nucleus is challenging because deep learning (DL) models pretrained using ImageNet perform poorly when applied directly to medical data due to the limited availability of labeled medical images and high interclass similarity. Self-supervised pretraining offers a solution by circumventing the need for cost-intensive data annotations and bridging domain disparities. In this study, to address the challenges of intelligent grading, we proposed a hybrid model called nuclear cataract mask encoder network (NCME-Net), which utilizes self-supervised pretraining for the four-class analysis of nuclear cataract severity. A total of 792 images of nuclear cataracts were categorized into the training set (533 images), the validation set (139 images), and the test set (100 images). NCME-Net achieved a diagnostic accuracy of 91.0 % on the test set, a 5.0 % improvement over the best-performing DL model (ResNet50). Experimental results demonstrate NCME-Net's ability to distinguish between cataract severities, particularly in scenarios with limited samples, making it a valuable tool for intelligently diagnosing cataracts. In addition, the effect of different self-supervised tasks on the model's ability to capture the intrinsic structure of the data was studied. Findings indicate that image restoration tasks significantly enhance semantic information extraction.

UTMGAT: a unified transformer with memory encoder and graph attention networks for multidomain dialogue state tracking

UTMGAT: a unified transformer with memory encoder and graph attention networks for multidomain dialogue state tracking

Nighttime road scene image enhancement based on cycle-consistent generative adversarial network

During nighttime road scenes, images are often affected by contrast distortion, loss of detailed information, and a significant amount of noise. These factors can negatively impact the accuracy of segmentation and object detection in nighttime road scenes. A cycle-consistent generative adversarial network has been proposed to address this issue to improve the quality of nighttime road scene images. The network includes two generative networks with identical structures and two adversarial networks with identical structures. The generative network comprises an encoder network and a corresponding decoder network. A context feature extraction module is designed as the foundational element of the encoder-decoder network to capture more contextual semantic information with different receptive fields. A receptive field residual module is also designed to increase the receptive field in the encoder network.The illumination attention module is inserted between the encoder and decoder to transfer critical features extracted by the encoder to the decoder. The network also includes a multiscale discriminative network to discriminate better whether the image is a real high-quality or generated image. Additionally, an improved loss function is proposed to enhance the efficacy of image enhancement. Compared to state-of-the-art methods, the proposed approach achieves the highest performance in enhancing nighttime images, making them clearer and more natural.

SSM-Net: Semi-supervised multi-task network for joint lesion segmentation and classification from pancreatic EUS images

Pancreatic cancer does not show specific symptoms, which makes the diagnosis of early stages difficult with established image-based screening methods and therefore has the worst prognosis among all cancers. Although endoscopic ultrasonography (EUS) has a key role in diagnostic algorithms for pancreatic diseases, B-mode imaging of the pancreas can be affected by confounders such as chronic pancreatitis, which can make both pancreatic lesion segmentation and classification laborious and highly specialized. To address these challenges, this work proposes a semi-supervised multi-task network (SSM-Net) to leverage unlabeled and labeled EUS images for joint pancreatic lesion classification and segmentation. Specifically, we first devise a saliency-aware representation learning module (SRLM) on a large number of unlabeled images to train a feature extraction encoder network for labeled images by computing a contrastive loss with a semantic saliency map, which is obtained by our spectral residual module (SRM). Moreover, for labeled EUS images, we devise channel attention blocks (CABs) to refine the features extracted from the pre-trained encoder on unlabeled images for segmenting lesions, and then devise a merged global attention module (MGAM) and a feature similarity loss (FSL) for obtaining a lesion classification result. We collect a large-scale EUS-based pancreas image dataset (LS-EUSPI) consisting of 9,555 pathologically proven labeled EUS images (499 patients from four categories) and 15,500 unlabeled EUS images. Experimental results on the LS-EUSPI dataset and a public thyroid gland lesion dataset show that our SSM-Net clearly outperforms state-of-the-art methods.

A multi-feature spatial–temporal fusion network for traffic flow prediction

The traffic flow prediction is the key to alleviate traffic congestion, yet very challenging due to the complex influence factors. Currently, the most of deep learning models are designed to dig out the intricate dependency in continuous standardized sequences, which are dependent to high requirements for data continuity and regularized distribution. However, the data discontinuity and irregular distribution are inevitable in the real-world practical application, then we need find a way to utilize the powerful effect of the multi-feature fusion rather than continuous relation in standardized sequences. To this end, we conduct the prediction based on the multiple traffic features reflecting the complex influence factors. Firstly, we propose the ATFEM, an adaptive traffic features extraction mechanism, which can select important influence factors to construct joint temporal features matrix and global spatial features matrix according to the traffic condition. In this way, the feature’s representation ability can be improved. Secondly, we propose the MFSTN, a multi-feature spatial–temporal fusion network, which include the temporal transformer encoder and graph attention network to obtain the latent representation of spatial–temporal features. Especially, we design the scaled spatial–temporal fusion module, which can automatically learn optimal fusion weights, further adapt to inconsistent spatial–temporal dimensions. Finally, the multi-layer perceptron gets the mapping function between these comprehensive features and traffic flow. This method helps to improve the interpretability of the prediction. Experimental results show that the proposed model outperforms a variety of baselines, and it can accurately predict the traffic flow when the data missing rate is high.

Multi-stream domain adversarial prototype network for integrated smart roller TBM main bearing fault diagnosis across various low rotating speeds

The well-being of tunnel boring machines (TBM) heavily relies on the health status of main bearings, significantly impacting both safety and operational efficiency. Investigating the operation monitoring and diagnosis algorithms of TBM main bearings in practical scenarios with limited fault data and real-time variations in low-speed heavy-load operating conditions poses a research challenge. To solve these issues, a novel domain-adversarial prototype network diagnostic algorithm with a multi-stream fusion feature encoder (MDAPN) based on smart roller monitoring data is proposed. In contrast to existing external vibration monitoring schemes for bearings, monitoring data from smart rollers can mitigate complex external interference and greatly shorten the transmission path of fault sources. The algorithm encompasses a multi-stream fusion feature encoder, domain discriminator, and prototype network classifier. Specifically, the multi-stream fusion feature encoder adopts the two-stream convolutional network to deeply extract the axial-radial fused features of rollers, incorporating shallow statistical feature information. Domain-invariant features are generated based on domain adversarial learning strategies, and the prototype network classifier reduces dependence on target domain samples. An integrated smart roller main bearing fault simulation testbed was built, conducting 6 sets of cross-domain experiments. The proposed method reached an average accuracy of 98.41 % under 5-shot, surpassing the baseline method by 6.57 % at least. This validates that the MDAPN based on smart roller state exhibits excellent fault recognition performance for the main bearing under varying operational conditions with scarce available samples.

FAGD-Net: Feature-Augmented Grasp Detection Network Based on Efficient Multi-Scale Attention and Fusion Mechanisms

Grasping robots always confront challenges such as uncertainties in object size, orientation, and type, necessitating effective feature augmentation to improve grasping detection performance. However, many prior studies inadequately emphasize grasp-related features, resulting in suboptimal grasping performance. To address this limitation, this paper proposes a new grasping approach termed the Feature-Augmented Grasp Detection Network (FAGD-Net). The proposed network incorporates two modules designed to enhance spatial information features and multi-scale features. Firstly, we introduce the Residual Efficient Multi-Scale Attention (Res-EMA) module, which effectively adjusts the importance of feature channels while preserving precise spatial information within those channels. Additionally, we present a Feature Fusion Pyramidal Module (FFPM) that serves as an intermediary between the encoder and decoder, effectively addressing potential oversights or losses of grasp-related features as the encoder network deepens. As a result, FAGD-Net achieved advanced levels of grasping accuracy, with 98.9% and 96.5% on the Cornell and Jacquard datasets, respectively. The grasp detection model was deployed on a physical robot for real-world grasping experiments, where we conducted a series of trials in diverse scenarios. In these experiments, we randomly selected various unknown household items and adversarial objects. Remarkably, we achieved high success rates, with a 95.0% success rate for single-object household items, 93.3% for multi-object scenarios, and 91.0% for cluttered scenes.

BP-MoE: Behavior Pattern-aware Mixture-of-Experts for Temporal Graph Representation Learning

Temporal graph representation learning aims to develop low-dimensional embeddings for nodes in a graph that can effectively capture their structural and temporal properties. Prior approaches primarily focus on devising sophisticated neural architectures to encode the historical interactions of nodes, enabling capturing the specific behavior patterns. However, such methods often overlook the fact that different nodes in a graph may exhibit distinct evolutionary preferences, resulting in a failure to adequately model the variations in the node evolution behavior. To address these issues, we propose a novel temporal graph learning framework called Behavior Pattern-aware Mixture-of-Experts (BP-MoE), which leverages multiple expert encoder networks with diverse architectures to comprehensively capture the behavior patterns for nodes. In detail, we first design a multi-perspective graph encoder that employs experts based on the gated recurrent units, graph attention units, and multilayer perceptron units to encode a node’s long-term memory, high-order neighborhood features, and short-term behavior preference, respectively. Subsequently, to achieve the personalized evolution preference learning for each individual node, we incorporate a behavior pattern-aware MoE mechanism to activate the pattern-specific experts for each node via a gating network. Finally, to optimize the training process of expert networks, we introduce two auxiliary losses that prevent the imbalanced training across experts. Extensive experiments conducted on multiple benchmark datasets verify the superiority of our approach over the state-of-the-art baselines. Specifically, BP-MoE consistently outperforms the baseline models for the link prediction and node classification tasks under various experimental settings.

BERT4FCA: A method for bipartite link prediction using formal concept analysis and BERT.

Link prediction in bipartite networks finds practical applications in various domains, including friend recommendation in social networks and chemical reaction prediction in metabolic networks. Recent studies have highlighted the potential for link prediction by maximal bi-cliques, which is a structural feature within bipartite networks that can be extracted using formal concept analysis (FCA). Although previous FCA-based methods for bipartite link prediction have achieved good performance, they still have the problem that they cannot fully capture the information of maximal bi-cliques. To solve this problem, we propose a novel method for link prediction in bipartite networks, utilizing a BERT-like transformer encoder network to enhance the contribution of FCA to link prediction. Our method facilitates bipartite link prediction by learning more information from the maximal bi-cliques and their order relations extracted by FCA. Experimental results on five real-world bipartite networks demonstrate that our method outperforms previous FCA-based methods, a state-of-the-art Graph Neural Network(GNN)-based method, and classic methods such as matrix-factorization and node2vec.

Higher-Order Convolutional Neural Networks for Essential Climate Variables Forecasting

Earth observation imaging technologies, particularly multispectral sensors, produce extensive high-dimensional data over time, thus offering a wealth of information on global dynamics. These data encapsulate crucial information in essential climate variables, such as varying levels of soil moisture and temperature. However, current cutting-edge machine learning models, including deep learning ones, often overlook the treasure trove of multidimensional data, thus analyzing each variable in isolation and losing critical interconnected information. In our study, we enhance conventional convolutional neural network models, specifically those based on the embedded temporal convolutional network framework, thus transforming them into models that inherently understand and interpret multidimensional correlations and dependencies. This transformation involves recasting the existing problem as a generalized case of N-dimensional observation analysis, which is followed by deriving essential forward and backward pass equations through tensor decompositions and compounded convolutions. Consequently, we adapt integral components of established embedded temporal convolutional network models, like encoder and decoder networks, thus enabling them to process 4D spatial time series data that encompass all essential climate variables concurrently. Through the rigorous exploration of diverse model architectures and an extensive evaluation of their forecasting prowess against top-tier methods, we utilize two new, long-term essential climate variables datasets with monthly intervals extending over four decades. Our empirical scrutiny, particularly focusing on soil temperature data, unveils that the innovative high-dimensional embedded temporal convolutional network model-centric approaches markedly excel in forecasting, thus surpassing their low-dimensional counterparts, even under the most challenging conditions characterized by a notable paucity of training data.

DEAttentionDTA: protein-ligand binding affinity prediction based on dynamic embedding and self-attention.

Predicting protein-ligand binding affinity is crucial in new drug discovery and development. However, most existing models rely on acquiring 3D structures of elusive proteins. Combining amino acid sequences with ligand sequences and better highlighting active sites are also significant challenges. We propose an innovative neural network model called DEAttentionDTA, based on dynamic word embeddings and a self-attention mechanism, for predicting protein-ligand binding affinity. DEAttentionDTA takes the 1D sequence information of proteins as input, including the global sequence features of amino acids, local features of the active pocket site, and linear representation information of the ligand molecule in the SMILE format. These three linear sequences are fed into a dynamic word-embedding layer based on a 1D convolutional neural network for embedding encoding and are correlated through a self-attention mechanism. The output affinity prediction values are generated using a linear layer. We compared DEAttentionDTA with various mainstream tools and achieved significantly superior results on the same dataset. We then assessed the performance of this model in the p38 protein family. The resource codes are available at https://github.com/whatamazing1/DEAttentionDTA.

DualTrans: A Novel Glioma Segmentation Framework Based on a Dual-Path Encoder Network and Multi-View Dynamic Fusion Model

Segmentation methods based on convolutional neural networks (CNN) have achieved remarkable results in the field of medical image segmentation due to their powerful representation capabilities. However, for brain-tumor segmentation, owing to the significant variations in shape, texture, and location, traditional convolutional neural networks (CNNs) with limited convolutional kernel-receptive fields struggle to model explicit long-range (global) dependencies, thereby restricting segmentation accuracy and making it difficult to accurately identify tumor boundaries in medical imaging. As a result, researchers have introduced the Swin Transformer, which has the capability to model long-distance dependencies, into the field of brain-tumor segmentation, offering unique advantages in the global modeling and semantic interaction of remote information. However, due to the high computational complexity of the Swin Transformer and its reliance on large-scale pretraining, it faces constraints when processing large-scale medical images. Therefore, this study addresses this issue by proposing a smaller network, consisting of a dual-encoder network, which also resolves the instability issue that arises in the training process of large-scale visual models with the Swin Transformer, where activation values of residual units accumulate layer by layer, leading to a significant increase in differences in activation amplitudes across layers and causing model instability. The results of the experimental validation using real data show that our dual-encoder network has achieved significant performance improvements, and it also demonstrates a strong appeal in reducing computational complexity.

Improved fault detection and diagnosis using graph auto encoder and attention-based graph convolution networks

A powerful fault detection and diagnosis (FDD) system plays a pivotal role in achieving operational excellence by maximizing system performance, optimizing maintenance strategies, and ensuring the longevity and resilience of process plants. In the context of FDD for multivariate sensor data, this study presents an improved FDD approach using graph-based neural networks. This graph neural network uses an adjacency matrix developed by extracting the expert domain knowledge and topological information of the multi-sensor system. This additional graph representation of the system is incorporated along with multivariate sensor data to capture the spatial and temporal information in neural networks efficiently. In this regard, we propose and evaluate: 1) A Graph Auto Encoder (GAE) based fault detection strategy and 2) An Attention-based Spatial Temporal Graph Convolution Network (ASTGCN) based fault diagnosis methodology. By leveraging the additional knowledge in the form of graphs, the GAE captures the complex relationships and dependencies among sensors, enabling effective anomaly detection, which identifies abnormal patterns and deviations from normal behavior, thus indicating potential faults in the system. The ASTGCN incorporates attention mechanisms to selectively focus on relevant sensor nodes and capture their spatial and temporal dependencies for fault diagnosis. The effectiveness of the proposed FDD approach is demonstrated using the benchmark Tennessee Eastman Process (TEP) problem. The results show that the proposed approaches outperform traditional methods and highlight the importance of leveraging graph-based knowledge for FDD in complex systems.

An Efficient Method for Detecting Abnormal Electricity Behavior

The non-technical losses caused by abnormal power consumption behavior of power users seriously affect the revenue of power companies and the quality of power supply. To assist electric power companies in improving the efficiency of power consumption audit and regulating the power consumption behavior of users, this paper proposes a power consumption anomaly detection method named High-LowDAAE (Autoencoder model for dual adversarial training of high low-level temporal features). High-LowDAAE adds an extra “discriminator” named AE3 to USAD (UnSupervised Anomaly Detection on Multivariate Time Series), which performs the same function as AE2 in USAD. AE3 performs the same function as AE2 in USAD, i.e., it is trained against AE1 to enhance its ability to reconstruct average data. However, AE3 differs from AE2 because the two “discriminators” correspond to the high-level and low-level time series features output from the shared encoder network. By utilizing different levels of temporal features to reconstruct the data and conducting adversarial training, AE1 can reconstruct the time-series data more efficiently, thus improving the accuracy of detecting abnormal electricity usage. In addition, to enhance the model’s feature extraction ability for time-series data, the self-encoder is constructed with a long short-term memory (LSTM) network, and the fully connected layer in the USAD model is no longer used. This modification improves the extraction of temporal features and provides richer hidden features for the adversarial training of the dual “discriminators”. Finally, the ablation and comparison experiments are conducted using accurate electricity consumption data from users, and the results show that the proposed method has higher accuracy.

Concrete spalling detection system based on semantic segmentation using deep architectures

This paper presents a method for detecting the location of spalling and assessing the severity level of the spalling in concrete surfaces. The proposed method is constructed based on deep learning architectures and multi-class semantic segmentation. The proposed method can detect each pixel as a non-spalling, a deep-spalling, or a shallow-spalling. The proposed method consists of three different deep learning architectures with several encoders as backbone networks. Both qualitative and quantitative analyses show that the deep learning architecture with a certain encoder network can detect spalling with different severity levels very well. Additionally, the paper proposes a method to analyze the deep spalling areas of concrete to show their severity levels. The performance analysis shows that this approach provides very convincing results with respect to the actual affected spalling areas. The results convey that this paper achieved a higher level of performance for detecting spalling and assessing the severity of the spalling.

Robust-DSN: A Hybrid Distributed Replication and Encoding Network Grouped with a Distributed Swarm Workflow Scheduler

In many distributed applications such as the Internet of Things (IoT), large amounts of data are being generated that require robust storage solutions. Traditional cloud solutions, although efficient, often lack trust and transparency because of centralized management. To address these issues, we present Robust-DSN, a distributed storage network leveraging the hybrid distributed replication and encoding network (HYDREN) and the distributed swarm workflow scheduler (DSWS) as its main components. Our system uses an interplanetary file system (IPFS) as an underlay storage network and segments it into multiple regions to distribute the failure domain and improve the data’s proximity to users. HYDREN incorporates Reed–Solomon encoding and distributed replication to improve file availability, while DSWS optimizes resource allocation across the network. The uploaded file is encoded into chunks and distributed across distinct optimal nodes leveraging lightweight multithreading. Additionally, Robust-DSN verifies the integrity of all chunks by preserving the hashes when uploading and validating each chunk while downloading. The proposed system provides a comprehensive solution for resilient distributed data storage, focusing on the key challenges of data availability, integrity, and performance. The results reveal that compared with a state-of-the-art system, the proposed system improves file recovery by 15%, even with a 50% peer failure rate. Furthermore, with replication factor 4 and the same failure resilience as IPFS, it saves 50% storage and enhances file recovery by 8%. Robust-DSN acts as a distributed storage platform for emerging technologies, expanding storage system capabilities in a wide range of distributed applications.