Decoding Framework Research Articles

A Review of Deep Learning-Based Remote Sensing Image Caption: Methods, Models, Comparisons and Future Directions

Remote sensing images contain a wealth of Earth-observation information. Efficient extraction and application of hidden knowledge from these images will greatly promote the development of resource and environment monitoring, urban planning and other related fields. Remote sensing image caption (RSIC) involves obtaining textual descriptions from remote sensing images through accurately capturing and describing the semantic-level relationships between objects and attributes in the images. However, there is currently no comprehensive review summarizing the progress in RSIC based on deep learning. After defining the scope of the papers to be discussed and summarizing them all, the paper begins by providing a comprehensive review of the recent advancements in RSIC, covering six key aspects: encoder–decoder framework, attention mechanism, reinforcement learning, learning with auxiliary task, large visual language models and few-shot learning. Subsequently a brief explanation on the datasets and evaluation metrics for RSIC is given. Furthermore, we compare and analyze the results of the latest models and the pros and cons of different deep learning methods. Lastly, future directions of RSIC are suggested. The primary objective of this review is to offer researchers a more profound understanding of RSIC.

A Novel RUL-Centric Data Augmentation Method for Predicting the Remaining Useful Life of Bearings

Maintaining the reliability of rotating machinery in industrial environments entails significant challenges. The objective of this paper is to develop a methodology that can accurately predict the condition of rotating machinery in order to facilitate the implementation of effective preventive maintenance strategies. This article proposed a novel RUL-centric data augmentation method, designated as DF-MDAGRU, for the purpose of predicting the remaining useful life (RUL) of bearings. This model is based on an encoder–decoder framework that integrates time–frequency domain feature enhancement with multidimensional dynamic attention gated recurrent units for feature extraction. This method enhances time–frequency domain features through the Discrete Wavelet Downsampling module (DWD) and Convolutional Fourier Residual Block (CFRB). This method employs a Multiscale Channel Attention Module (MS-CAM) and a Multiscale Convolutional Spatial Attention Mechanism (MSSAM) to extract channel and spatial feature information. Finally, the output predictions are processed through linear regression to achieve the final RUL estimation. Experimental results demonstrate that the proposed method outperforms other state-of-the-art approaches on the FEMETO-ST and XJTU datasets.

A Decentralized Digital Watermarking Framework for Secure and Auditable Video Data in Smart Vehicular Networks

Thanks to the rapid advancements in Connected and Automated Vehicles (CAVs) and vehicular communication technologies, the concept of the Internet of Vehicles (IoVs) combined with Artificial Intelligence (AI) and big data promotes the vision of an Intelligent Transportation System (ITS). An ITS is critical in enhancing road safety, traffic efficiency, and the overall driving experience by enabling a comprehensive data exchange platform. However, the open and dynamic nature of IoV networks brings significant performance and security challenges to IoV data acquisition, storage, and usage. To comprehensively tackle these challenges, this paper proposes a Decentralized Digital Watermarking framework for smart Vehicular networks (D2WaVe). D2WaVe consists of two core components: FIAE-GAN, a novel feature-integrated and attention-enhanced robust image watermarking model based on a Generative Adversarial Network (GAN), and BloVA, a Blockchain-based Video frames Authentication scheme. By leveraging an encoder–noise–decoder framework, trained FIAE-GAN watermarking models can achieve the invisibility and robustness of watermarks that can be embedded in video frames to verify the authenticity of video data. BloVA ensures the integrity and auditability of IoV data in the storing and sharing stages. Experimental results based on a proof-of-concept prototype implementation validate the feasibility and effectiveness of our D2WaVe scheme for securing and auditing video data exchange in smart vehicular networks.

Dual Stream Encoder–Decoder Architecture with Feature Fusion Model for Underwater Object Detection

Underwater surveillance is an imminent and fascinating exploratory domain, particularly in monitoring aquatic ecosystems. This field offers valuable insights into underwater behavior and activities, which have broad applications across various domains. Specifically, underwater surveillance involves detecting and tracking moving objects within aquatic environments. However, the complex properties of water make object detection a challenging task. Background subtraction is a commonly employed technique for detecting local changes in video scenes by segmenting images into the background and foreground to isolate the object of interest. Within this context, we propose an innovative dual-stream encoder–decoder framework based on the VGG-16 and ResNet-50 models for detecting moving objects in underwater frames. The network includes a feature fusion module that effectively extracts multiple-level features. Using a limited set of images and performing training in an end-to-end manner, the proposed framework yields accurate results without post-processing. The efficacy of the proposed technique is confirmed through visual and quantitative comparisons with eight cutting-edge methods using two standard databases. The first one employed in our experiments is the Underwater Change Detection Dataset, which includes five challenges, each challenge comprising approximately 1000 frames. The categories in this dataset were recorded under various underwater conditions. The second dataset used for practical analysis is the Fish4Knowledge dataset, where we considered five challenges. Each category, recorded in different aquatic settings, contains a varying number of frames, typically exceeding 1000 per category. Our proposed method surpasses all methods used for comparison by attaining an average F-measure of 0.98 on the Underwater Change Detection Dataset and 0.89 on the Fish4Knowledge dataset.

A Bi‐level stacked LSTM‐DNN‐based decoder network for AGC dispatch under regulation market framework in presence of VPP and EV aggregators

AbstractThe consideration of mileage settlement in the frequency regulation market has encouraged fast‐acting units, such as converter‐interfaced generators (CIG) and electric vehicle stations, to actively participate in load‐generation balancing through automatic generation control (AGC). Conventional frequency regulation faces challenges in coping with the growing variability of CIGs and also lacks effective incentives for rapid‐responding units. In this context, a bi‐level AGC dispatch approach based on a stacked long short‐term memory (LSTM)‐deep neural network (DNN)‐based decoder framework is proposed for a power system comprising diverse CIGs forming a virtual power plant and electric vehicle aggregators. The proposed decoder network is comprised of stacked LSTM and DNN, wherein the cascaded LSTM layers are introduced to accurately capture temporal information from time series input. The inclusion of a dropout mechanism further enhances the model’s generalisability in unforeseen environments. The proposed dispatch framework uses mileage‐based compensation criteria to optimally allocate instructions among various participating units with differing regulation characteristics. The performance of the proposed method is analysed by considering packet loss, delay, unexpected generation failure, and denial of service attacks. The evaluation of the proposed approach reveals its superior performance compared to proportionality, particle swarm optimisation, decision tree, and DNN methods.

An Enhanced Transportation System for People of Determination.

Visually Impaired Persons (VIPs) have difficulty in recognizing vehicles used for navigation. Additionally, they may not be able to identify the bus to their desired destination. However, the bus bay in which the designated bus stops has not been analyzed in the existing literature. Thus, a guidance system for VIPs that identifies the correct bus for transportation is presented in this paper. Initially, speech data indicating the VIP's destination are pre-processed and converted to text. Next, utilizing the Arctan Gradient-activated Recurrent Neural Network (ArcGRNN) model, the number of bays at the location is detected with the help of a Global Positioning System (GPS), input text, and bay location details. Then, the optimal bay is chosen from the detected bays by utilizing the Experienced Perturbed Bacteria Foraging Triangular Optimization Algorithm (EPBFTOA), and an image of the selected bay is captured and pre-processed. Next, the bus is identified utilizing a You Only Look Once (YOLO) series model. Utilizing the Sub-pixel Shuffling Convoluted Encoder-ArcGRNN Decoder (SSCEAD) framework, the text is detected and segmented for the buses identified in the image. From the segmented output, the text is extracted, based on the destination and route of the bus. Finally, regarding the similarity value with respect to the VIP's destination, a decision is made utilizing the Multi-characteristic Non-linear S-Curve-Fuzzy Rule (MNC-FR). This decision informs the bus conductor about the VIP, such that the bus can be stopped appropriately to pick them up. During testing, the proposed system selected the optimal bay in 247,891 ms, which led to deciding the bus stop for the VIP with a fuzzification time of 34,197 ms. Thus, the proposed model exhibits superior performance over those utilized in prevailing works.

An fMRI-based visual decoding framework combined with two-stage learning and asynchronous iterative update strategy

An fMRI-based visual decoding framework combined with two-stage learning and asynchronous iterative update strategy

Retrieval In Decoder benefits generative models for explainable complex question answering

Large-scale Language Models (LLMs) utilizing the Chain-of-Thought prompting demonstrate exceptional performance in a variety of tasks. However, the persistence of factual hallucinations remains a significant challenge in practical applications. Prevailing retrieval-augmented methods treat the retriever and generator as separate components, which inadvertently restricts the generator’s capabilities to those of the retriever through intensive supervised training. In this work, we propose an unsupervised Retrieval In Decoder framework for multi-granularity decoding called RID, which integrates retrieval directly into the decoding process of generative models. It dynamically adjusts decoding granularity based on retrieval outcomes, and duly corrects the decoding direction through its direct impact on the next token. Moreover, we introduce a reinforcement learning-driven knowledge distillation method for adaptive explanation generation to better apply to Small-scale Language Models (SLMs). The experimental results across six public benchmarks surpass popular LLMs and existing retrieval-augmented methods, which demonstrates the effectiveness of RID in models of different scales and verifies its applicability and scalability.

A novel multiscale cGAN approach for enhanced salient object detection in single haze images

In computer vision, image dehazing is a low-level task that employs algorithms to analyze and remove haze from images, resulting in haze-free visuals. The aim of Salient Object Detection (SOD) is to locate the most visually prominent areas in images. However, most SOD techniques applied to visible images struggle in complex scenarios characterized by similarities between the foreground and background, cluttered backgrounds, adverse weather conditions, and low lighting. Identifying objects in hazy images is challenging due to the degradation of visibility caused by atmospheric conditions, leading to diminished visibility and reduced contrast. This paper introduces an innovative approach called Dehaze-SOD, a unique integrated model that addresses two vital tasks: dehazing and salient object detection. The key novelty of Dehaze-SOD lies in its dual functionality, seamlessly integrating dehazing and salient object identification into a unified framework. This is achieved using a conditional Generative Adversarial Network (cGAN) comprising two distinct subnetworks: one for image dehazing and another for salient object detection. The first module, designed with residual blocks, Dark Channel Prior (DCP), total variation, and the multiscale Retinex algorithm, processes the input hazy images. The second module employs an enhanced EfficientNet architecture with added attention mechanisms and pixel-wise refinement to further improve the dehazing process. The outputs from these subnetworks are combined to produce dehazed images, which are then fed into our proposed encoder–decoder framework for salient object detection. The cGAN is trained with two modules working together: the generator aims to produce haze-free images, whereas the discriminator distinguishes between the generated haze-free images and real haze-free images. Dehaze-SOD demonstrates superior performance compared to state-of-the-art dehazing methods in terms of color fidelity, visibility enhancement, and haze removal. The proposed method effectively produces high-quality, haze-free images from various hazy inputs and accurately detects salient objects within them. This makes Dehaze-SOD a promising tool for improving salient object detection in challenging hazy conditions. The effectiveness of our approach has been validated using benchmark evaluation metrics such as mean absolute error (MAE), peak signal-to-noise ratio (PSNR), and structural similarity index measure (SSIM).

Scene Chinese Recognition with Local and Global Attention

Recently, scene Chinese recognition has attracted increasing attention. While mainstream scene text recognition methods exhibit outstanding performance in English recognition, they are considerably limited in Chinese recognition, due to inter-class similarity, intra-class variability, and complex combination of components in scene Chinese text. In this paper, we design Adaptive Position Encoding(APE) to enhance the model’s ability to perceive spatial information. Based on APE, we have innovatively designed Local Attention Module (LAM) and Global Attention Module (GAM). Specifically, LAM captures local features to identify common characteristics among characters of the same category, addressing the issue of intra-class variability. Meanwhile, LAM captures global features to identify the subordination relationships of Chinese character components. By integrating LAM and GAM, combining both local and global features, it is possible to find differences in the details among features that are fundamentally similar, thus solving the problem of inter-class similarity. Further, we contrive the transformer encoder–decoder structure to identify the vast variety of Chinese characters. Based on the Local/Global Attention Module and transformer encoder–decoder framework, we devise the novel sequence-to-sequence Local and Global Attention Network(LGANet), where both the backbone and the encoder/decoder are composed of attention mechanisms. Subsequent experiments on the Chinese scene dataset show that the recognition accuracy of our proposed LGANet is 77.3% and the normalized editing distance is 88.6%, both of which achieve the SOTA results in Fig. 1.

Attribute Diversity Aware Community Detection on Attributed Graphs Using Three-View Graph Attention Neural Networks

Community detection is a fundamental yet important task for characterizing and understanding the structure of attributed graphs. Existing methods mainly focus on the structural tightness and attribute similarity among nodes in a community. However, grouping numerous semantically homogeneous nodes will result in information cocoons and thus reduce the robustness of community structure and the efficiency of node collaboration in real-world applications, such as recommendation systems and collaboration networks. Since nodes with closer connections tend to be more similar, finding communities with dense structures and diverse attributes poses great challenges to mining latent relationships between the graph structure and attribute distribution. To our best knowledge, very little research has been conducted to address this challenge. In this article, we propose a novel three-view graph attention neural networks (TvGANN) model to formally address the attribute diversity aware community detection problem. TvGANN reveals correlations between the graph structure and attributes distribution from the perspective of node organization, attribute co-occurrence, and the node-attribute interaction. It effectively captures structural features and attributes distribution by feeding a structural network and an attribute co-occurrence network into graph attention modules through the encoder–decoder framework. It also learns heterogeneous information by feeding a network into a meta-node attention module. Then, it fuzes the three modules and clusters the embedding representations through a Student's t -distribution approach, which iteratively refines the clustering results. The experiments show that our method not only improves the quality in dense community detection but also performs efficiently for attributed graphs.

Physics‐Informed Convolutional Decoder (PICD): A Novel Approach for Direct Inversion of Heterogeneous Subsurface Flow

AbstractWe propose a physics‐informed convolutional decoder (PICD) framework for inverse modeling of heterogenous groundwater flow. PICD stands out as a direct inversion method, eliminating the need for repeated forward model simulations. The framework combines data‐driven and physics‐driven approaches by integrating monitoring data and domain knowledge into the inversion process. PICD utilizes a convolutional decoder to effectively approximate the spatial distribution of hydraulic heads, while Karhunen–Loève expansion (KLE) is employed to parameterize hydraulic conductivities. During the training process, the stochastic vector in KLE and the parameters of the convolutional decoder are adjusted simultaneously to minimize the data‐mismatch and the physical violation. The final optimized stochastic vectors correspond to the estimation of hydraulic conductivities, and the trained convolutional decoder can predict the evolution and distribution of hydraulic heads. Various scenarios of groundwater flow are examined and results demonstrate the framework's capability to accurately estimate heterogeneous hydraulic conductivities and to deliver satisfactory predictions of hydraulic heads, even with sparse measurements.

DeConFCluster: Deep Convolutional Transform Learning based multiview clustering fusion framework

Multi-view data clustering is essential for discovering patterns and exploiting information from different sources. In this context, we propose DeConFCluster, an unsupervised multi-view clustering fusion framework based on Deep Convolutional Transform Learning (CTL). Our approach has the advantage that it does not require an additional decoder network during the training phase. This makes our model less prone to overfitting in data-constrained scenarios, as opposed to several recent studies based on the encoder–decoder framework. Furthermore, our method incorporates a loss function inspired by K-Means, which enables it to learn more effective representations for the clustering task. Finally, we evaluate our framework on five standard multi-view clustering datasets, and show that it outperforms the state-of-the-art multi-view deep clustering techniques.

An unsupervised deep global–local views model for anomaly detection in attributed networks

An attribute network is a form of data that contains rich semantic information. Many real scenarios can be modeled as attributed networks, such as social media, citations, and traffic networks. Anomaly detection in attributed networks is an interesting research topic owing to its potential in various practical applications, including spam, network intrusion, and financial fraud detection. However, attributed networks exhibit many anomaly patterns, such as structural, attribute, local, and global anomalies, making anomaly detection in attributed networks a challenging task. To address these difficulties, we designed DeepGL, a novel unsupervised deep global–local view model, for anomaly detection in attributed networks. Our model is an encoder–decoder framework with multiple views that capture node attributes and network structure information from both global and local views. Specifically, our model contains two encoders and four decoders. The two encoders are used to capture network features from local and global views, and the four decoders are used to reconstruct the local node attribute information, local structure information, global node attribute information, and global structure information. To the encoders and decoders, we applied Laplacian sharpening and smoothing techniques to maintain the integrity of normal node features while diminishing the conspicuousness of anomalous nodes in the reconstructed information, thereby facilitating the calculation of reconstruction errors. Extensive experiments on four real-world attributed network datasets demonstrate the excellent performance of the proposed method.

Non-autoregressive personalized bundle generation

The personalized bundle generation problem, which aims to create a preferred bundle for user from numerous candidate items, receives increasing attention in recommendation. However, existing works ignore the order-invariant nature of the bundle and adopt sequential modeling methods as the solution, which might introduce inductive bias and cause a large latency in prediction. To address this problem, we propose to perform the bundle generation via non-autoregressive mechanism and design a novel encoder–decoder framework named BundleNAT, which can effectively output the targeted bundle in one-shot without relying on any inherent order. In detail, instead of learning sequential dependency, we propose to adopt pre-training techniques and graph neural network to fully embed user-based preference and item-based compatibility information, and use a self-attention based encoder to further extract global dependency pattern. We then design a permutation-equivariant decoding architecture that is able to directly output the desired bundle in a one-shot manner. Experiments on three real-world datasets from Youshu and Netease show the proposed BundleNAT significantly outperforms the current state-of-the-art methods in average by up to 35.92%, 10.97% and 23.67% absolute improvements in Precision, Precision+, and Recall, respectively.

Application of Fashion Element Trend Prediction Model Integrating AM and EfficientNet-b7 Models in Art Design

With the rapid development of the fashion industry and the increasing diversification of consumer needs, accurately predicting the fashion trends of clothing elements has become an urgent problem in the field of art design. In order to solve these problems, this paper uses deep learning technology for fashion trend prediction and optimization of prediction models. Firstly, the EfficientNet-b7 model is constructed as an attribute predictor to accurately extract the attributes of clothing image elements. Then, based on user information, popular elements are grouped and counted to solve the problem of different populations having different opinions on popular trends. The prediction model is constructed based on the bidirectional long short-term memory network encoder decoder framework, which trains trend information as a whole and utilizes element coexistence relationships to assist in trend prediction. Meanwhile, the study uses random sampling method for clothing original adoption, and the experimental results show that the model considering coexistence relationship performs the best in terms of mean absolute error and mean absolute percentage error, which are 0.0132 and 14.68%, respectively. In addition, the model that introduces attention mechanism based on trend similarity has improved by 5.09% and 4.54% on two indicators compared to the latest model. The experimental results indicate that coexistence relationships can help improve the performance of prediction models. The attention mechanism based on trend similarity can further improve model performance by similarity comparison between historical information and changing trend, and selecting similar historical information as an important influencing factor for future trends.

Two-Level Battery Health Diagnosis Using Encoder–Decoder Framework and Gaussian Mixture Ensemble Learning Based on Relaxation Voltage

Two-Level Battery Health Diagnosis Using Encoder–Decoder Framework and Gaussian Mixture Ensemble Learning Based on Relaxation Voltage

Effective Video Summarization Using Channel Attention-Assisted Encoder–Decoder Framework

A significant number of cameras regularly generate massive amounts of data, demanding hardware, time, and labor resources to acquire, process, and monitor. Asymmetric frames within videos pose a challenge to automatic summarization of videos, making it challenging to capture key content. Developments in computer vision have accelerated the seamless capture and analysis of high-resolution video content. Video summarization (VS) has garnered considerable interest due to its ability to provide concise summaries of lengthy videos. The current literature mainly relies on a reduced set of representative features implemented using shallow sequential networks. Therefore, this work utilizes an optimal feature-assisted visual intelligence framework for representative feature selection and summarization. Initially, the empirical analysis of several features is performed, and ultimately, we adopt a fine-tuning InceptionV3 backbone for feature extraction, deviating from conventional approaches. Secondly, our strategic encoder–decoder module captures complex relationships with five convolutional blocks and two convolution transpose blocks. Thirdly, we introduced a channel attention mechanism, illuminating interrelations between channels and prioritizing essential patterns to grasp complex refinement features for final summary generation. Additionally, comprehensive experiments and ablation studies validate our framework's exceptional performance, consistently surpassing state-of-the-art networks on two benchmarks (TVSum and SumMe) datasets.

ABANet: Attention boundary‐aware network for image segmentation

AbstractDeep learning techniques have attained substantial progress in various face‐related tasks, such as face recognition, face inpainting, and facial expression recognition. To prevent infection or the spread of the virus, wearing of masks in public places has been mandated following the COVID‐19 epidemic, which has led to face occlusion and posed significant challenges for face recognition systems. Most prominent masked face recognition solutions rely on mask segmentation tasks. Therefore, segmentation can be used to mitigate the negative impacts of wearing a mask and improve recognition accuracy. Mask region segmentation suffers from two main problems: there is no standard type of masks that people wear, they come in different colours and designs, and there is no publicly available masked face dataset with appropriate ground truth for the mask region. In order to address these issues, we propose an encoder–decoder framework that utilizes a boundary‐aware attention network combined with a new hybrid loss to provide a map, patch, and pixel‐level supervision. We also introduce a dataset called MFSD, with 11,601 images and 12,758 masked faces for masked face segmentation. Furthermore, we compare the performance of different cutting‐edge deep learning semantic segmentation models on the presented dataset. Experimental results on the MSFD dataset reveal that the suggested approach outperforms state‐of‐the‐art, algorithms with 97.623% accuracy, 93.814% IoU, and 96.817% F1‐score rate. Our dataset of masked faces with mask region labels and source code will be available online.

Incorporating Entity Type-Aware and Word–Word Relation-Aware Attention in Generative Named Entity Recognition

Named entity recognition (NER) is a critical subtask in natural language processing. It is particularly valuable to gain a deeper understanding of entity boundaries and entity types when addressing the NER problem. Most previous sequential labeling models are task-specific, while recent years have witnessed the rise of generative models due to the advantage of tackling NER tasks in the encoder–decoder framework. Despite achieving promising performance, our pilot studies demonstrate that existing generative models are ineffective at detecting entity boundaries and estimating entity types. In this paper, a multiple attention framework is proposed which introduces the attention of entity-type embedding and word–word relation into the named entity recognition task. To improve the accuracy of entity-type mapping, we adopt an external knowledge base to calculate the prior entity-type distributions and then incorporate the information input to the model via the encoder’s self-attention. To enhance the contextual information, we take the entity types as part of the input. Our method obtains the other attention from the hidden states of entity types and utilizes it in self- and cross-attention mechanisms in the decoder. We transform the entity boundary information in the sequence into word–word relations and extract the corresponding embedding into the cross-attention mechanism. Through word–word relation information, the method can learn and understand more entity boundary information, thereby improving its entity recognition accuracy. We performed experiments on extensive NER benchmarks, including four flat and two long entity benchmarks. Our approach significantly improves or performs similarly to the best generative NER models. The experimental results demonstrate that our method can substantially enhance the capabilities of generative NER models.