Feature Fusion Strategy Research Articles

Ultra-short-term forecasting model of power load based on fusion of power spectral density and Morlet wavelet.

An accurate ultra-short-term time series prediction of a power load is an important guarantee for power dispatching and the safe operation of power systems. Problems of the current ultra-short-term time series prediction algorithms include low prediction accuracy, difficulty capturing the local mutation features, poor stability, and others. From the perspective of series decomposition, a multi-scale sequence decomposition model (TFDNet) based on power spectral density and the Morlet wavelet transform is proposed that combines the multidimensional correlation feature fusion strategy in the time and frequency domains. By introducing the time-frequency energy selection module, the "prior knowledge" guidance module, and the sequence denoising decomposition module, the model not only effectively delineates the global trend and local seasonal features, completes the in-depth information mining of the smooth trend and fluctuating seasonal features, but more importantly, realizes the accurate capture of the local mutation seasonal features. Finally, on the premise of improving the forecasting accuracy, single-point load forecasting and quantile probabilistic load forecasting for ultra-short-term load forecasting are realized. Through the experiments conducted on three public datasets and one private dataset, the TFDNet model reduces the mean square error (MSE) and mean absolute error (MAE) by 19.80 and 11.20% on average, respectively, as compared with the benchmark method. These results indicate the potential applications of the TFDNet model.

BLAM6A-Merge: Leveraging Attention Mechanisms and Feature Fusion Strategies to Improve the Identification of RNA N6-methyladenosine Sites.

RNA N6-methyladenosine is a prevalent and abundant type of RNA modification that exerts significant influence on diverse biological processes. To date, numerous computational approaches have been developed for predicting methylation, with most of them ignoring the correlations of different encoding strategies and failing to explore the adaptability of various attention mechanisms for methylation identification. To solve the above issues, we proposed an innovative framework for predicting RNA m6A modification site, termed BLAM6A-Merge. Specifically, it utilized a multimodal feature fusion strategy to combine the classification results of four features and Blastn tool. Apart from this, different attention mechanisms were employed for extracting higher-level features on specific features after the screening process. Extensive experiments on 12 benchmarking datasets demonstrated that BLAM6A-Merge achieved superior performance (average AUC: 0.849 for the full transcript mode and 0.784 for the mature mRNA mode). Notably, the Blastn tool was employed for the first time in the identification of methylation sites. The data and code can be accessed at https://github.com/DoraemonXia/BLAM6A-Merge.

The Use of Multi-Feature Fusion in the Evaluation of Emotional Expressions in Spoken English

Abstract Sentiment analysis is one of the important applications in the field of natural language processing. With the development of science and technology, sentiment analysis is developing in the direction of multi-feature fusion, and multi-feature fusion plays an important value in application in English spoken emotional expression. In this paper, we design a method for extracting multi-features based on multi-networks and propose a sentiment analysis model, ECISA-MFF model, on the basis of a multi-feature extraction framework and feature fusion scheme to solve the problem of data non-alignment and modal noise, and then further classify the sentiments and optimize the model. The article concludes with a comparison of the relevant performance of the models as well as a case study, and it is found that the F-macro value of the model proposed in this paper rises by 22.2% when compared with the BiLSTM+att+C model, which is significantly effective. After applying this paper’s method for emotion recognition, it was found that the average recognition rate of six emotions (happiness, anger, fear, surprise, sadness, and disgust) increased to 65.52%.

Adding Before Pruning: Sparse Filter Fusion for Deep Convolutional Neural Networks via Auxiliary Attention.

Filter pruning is a significant feature selection technique to shrink the existing feature fusion schemes (especially on convolution calculation and model size), which helps to develop more efficient feature fusion models while maintaining state-of-the-art performance. In addition, it reduces the storage and computation requirements of deep neural networks (DNNs) and accelerates the inference process dramatically. Existing methods mainly rely on manual constraints such as normalization to select the filters. A typical pipeline comprises two stages: first pruning the original neural network and then fine-tuning the pruned model. However, choosing a manual criterion can be somehow tricky and stochastic. Moreover, directly regularizing and modifying filters in the pipeline suffer from being sensitive to the choice of hyperparameters, thus making the pruning procedure less robust. To address these challenges, we propose to handle the filter pruning issue through one stage: using an attention-based architecture that adaptively fuses the filter selection with filter learning in a unified network. Specifically, we present a pruning method named adding before pruning (ABP) to make the model focus on the filters of higher significance by training instead of man-made criteria such as norm, rank, etc. First, we add an auxiliary attention layer into the original model and set the significance scores in this layer to be binary. Furthermore, to propagate the gradients in the auxiliary attention layer, we design a specific gradient estimator and prove its effectiveness for convergence in the graph flow through mathematical derivation. In the end, to relieve the dependence on the complicated prior knowledge for designing the thresholding criterion, we simultaneously prune and train the filters to automatically eliminate network redundancy with recoverability. Extensive experimental results on the two typical image classification benchmarks, CIFAR-10 and ILSVRC-2012, illustrate that the proposed approach performs favorably against previous state-of-the-art filter pruning algorithms.

Human Gait Recognition Based on Frontal-View Sequences Using Gait Dynamics and Deep Learning

To facilitate human gait recognition, this paper proposes a new frontal-view gait recognition method using gait dynamics and deep learning. Rather than adopting lateral-view parameters as gait features in the literature, we employ improved frontal-view features and classification methods to avoid the recognition rate drops due to the complicated surveillance environment. Specifically, we characterize the binary walking silhouettes with three different kinds of frontal-view gait features, including kinematic features, spatial ratio features and area features. In addition, we capture the gait dynamics underlying the time-varying gait features to reflect temporal dynamics information of human walking. Furthermore, we incorporate the deep feature learning information into the recognition procedure to take advantage of the deep learning technique. To obtain the optimal recognition accuracy and robustness performance against walking condition variations, we calculate the similarity between the appearing test gait dynamics and the trained gait dynamics, and propose an error-based feature fusion scheme for gait recognition. To validate the efficacy of the proposed method, we conduct experiments on published gait databases by comparing with other existing gait recognition methods.

Interference Response Prediction of Receiver Based on Wavelet Transform and a Temporal Convolution Network

In order to improve the prediction performance of existing methods amidst multi modulation coupling interference in complex electromagnetic environments, this paper introduces a novel approach that integrates wavelet transform with a temporal convolutional network. The model begins with a data preprocessing stage, where wavelet transform decomposes the original signal into various scales. This step generates scale coefficients across different frequency categories, effectively reducing the signal length. To enhance the model’s ability to capture long-term dependencies in time series data, temporal convolutional networks are employed for feature extraction. Moreover, the model’s performance is further refined by incorporating an attention mechanism-driven feature fusion strategy. This strategy methodically combines high and low frequency features along with local and global characteristics. The model’s efficacy is validated using a custom MATLAB dataset, with simulation results confirming a significant improvement in prediction accuracy.

CAFseg: A Semantic segmentation network with cross aggregation fusion strategy for RGB-thermal semantic segmentation

Semantic segmentation utilises the RGB-Thermal (RGB-T) source images with the capacity of provide pixel-level prediction for surrounding scenes in harsh imaging conditions. However, existing feature fusion strategies for RGB-T semantic segmentation networks generally fail to obtain the desirable fused features, and segmentation edge errors tend to occur in the segmentation results due to the difference between the features of multi-modalities. To this end, a novel semantic segmentation network with cross aggregation fusion strategy named CAFseg is proposed for RGB-T semantic segmentation in this study. Three feature extraction branches are included in the proposed CAFseg to supply sufficient source modality features. The light-weighted feature extraction backbones which integrated residual blocks and MobileViT blocks are developed for each feature extraction branch to accomplish optimal balance between feature extraction capacity and computational efficiency. The cross aggregation fusion strategy which consists of cross aggregation fusion architecture and adaptive aggregation feature fusion modules is innovatively designed for the proposed network to obtain fused features with abundant complementary information. Moreover, three independent triangle-shaped decoders are implemented for their correspondence feature extraction branch to make them achieve outstanding segmentation results separately, and the final segmentation results are obtained by fusing the segmentation results from each decoder, hence, further correcting the segmentation edge errors. Extensive ablation studies and comparative experiments have been conducted on two publicly available RGB-T semantic segmentation datasets. Experimental results validated the effectiveness of each component of the proposed CAFseg and demonstrated the superior performance of CAFseg over other state-of-the-art RGB-T semantic segmentation networks for RGB-T semantic segmentation.

MSER: Multimodal speech emotion recognition using cross-attention with deep fusion

In human–computer interaction (HCI) and especially speech signal processing, emotion recognition is one of the most important and challenging tasks due to multi-modality and limited data availability. Nowadays, an intelligent system is required for real-world applications to efficiently process and understand the speaker's emotional state and to enhance the analytical abilities to assist communication by a human-machine interface (HMI). Designing a reliable and robust Multimodal Speech Emotion Recognition (MSER) to efficiently recognize emotions through multi-modality such as speech and text is necessary. This paper propose a novel MSER model with a deep feature fusion technique using a multi-headed cross-attention mechanism. The proposed model utilize audio and text cues to predict the emotion label accordingly. Our proposed model process the raw speech signal and text by CNN and feeds to corresponding encoders for discriminative and semantic feature extractions. The cross-attention mechanism is applied to both features to enhance the interaction between text and audio cues by crossway to extract the most relevant information for emotion recognition. Finally, combining the region-wise weights from both encoders enables interaction among different layers and paths by the proposed deep feature fusion scheme. The authors evaluate the proposed system using the IEMOCAP and MELD datasets and conduct extensive experiments that obtain state-of-the-art (SOTA) results and show a 4.5% improved recognition rate, respectively. Our model secured a significant improvement over SOTA methods, which shows the robustness and effectiveness of the proposed MSER model.

FedAFR: Enhancing Federated Learning with adaptive feature reconstruction

Federated learning is a distributed machine learning method where clients train models on local data to ensure that data will not be transmitted to a central server, providing unique advantages in privacy protection. However, in real-world scenarios, data between different clients may be non-Independently and Identically Distributed (non-IID) and imbalanced, leading to discrepancies among local models and impacting the efficacy of global model aggregation. To tackle this issue, this paper proposes a novel framework, FedARF, designed to improve Federated Learning performance by adaptively reconstructing local features during training. FedARF offers a simple reconstruction module for aligning feature representations from various clients, thereby enhancing the generalization capability of cross-client aggregated models. Additionally, to better adapt the model to each client’s data distribution, FedARF employs an adaptive feature fusion strategy for a more effective blending of global and local model information, augmenting the model’s accuracy and generalization performance. Experimental results demonstrate that our proposed Federated Learning method significantly outperforms existing methods in variety image classification tasks, achieving faster model convergence and superior performance when dealing with non-IID data distributions.

MGFGNet: an automatic underwater acoustic target recognition method based on the multi-gradient flow global feature enhancement network

The recognition of underwater acoustic targets plays a crucial role in marine vessel monitoring. However, traditional underwater target recognition models suffer from limitations, including low recognition accuracy and slow prediction speed. To address these challenges, this article introduces a novel approach called the Multi-Gradient Flow Global Feature Enhancement Network (MGFGNet) for automatic recognition of underwater acoustic targets. Firstly, a new spectrogram feature fusion scheme is presented, effectively capturing both the physical and brain-inspired features of the acoustic signal. This fusion technique enhances the representation of underwater acoustic data, resulting in more accurate recognition results. Moreover, MGFGNet utilizes the multi-gradient flow network and incorporates a multi-dimensional feature enhancement technique to achieve fast and precise end-to-end recognition. Finally, a loss function is introduced to mitigate the influence of unbalanced data sets on model recognition performance using Taylor series. This further enhances model recognition performance. Experimental evaluations were conducted on the DeepShip dataset to assess the performance of our proposed method. The results demonstrate the superiority of MGFGNet, achieving a recognition rate of 99.1%, which significantly surpasses conventional methods. Furthermore, MGFGNet exhibits improved efficiency compared to the widely used ResNet18 model, reducing the parameter count by 51.28% and enhancing prediction speed by 33.9%. Additionally, we evaluated the generalization capability of our model using the ShipsEar dataset, where MGFGNet achieves a recognition rate of 99.5%, indicating its superior performance when applied to unbalanced data. The promising results obtained in this study highlight the potential of MGFGNet in practical applications.

A Small Intestinal Stromal Tumor Detection Method Based on an Attention Balance Feature Pyramid.

Small intestinal stromal tumor (SIST) is a common gastrointestinal tumor. Currently, SIST diagnosis relies on clinical radiologists reviewing CT images from medical imaging sensors. However, this method is inefficient and greatly affected by subjective factors. The automatic detection method for stromal tumors based on computer vision technology can better solve these problems. However, in CT images, SIST have different shapes and sizes, blurred edge texture, and little difference from surrounding normal tissues, which to a large extent challenges the use of computer vision technology for the automatic detection of stromal tumors. Furthermore, there are the following issues in the research on the detection and recognition of SIST. After analyzing mainstream target detection models on SIST data, it was discovered that there is an imbalance in the features at different levels during the feature fusion stage of the network model. Therefore, this paper proposes an algorithm, based on the attention balance feature pyramid (ABFP), for detecting SIST with unbalanced feature fusion in the target detection model. By combining weighted multi-level feature maps from the backbone network, the algorithm creates a balanced semantic feature map. Spatial attention and channel attention modules are then introduced to enhance this map. In the feature fusion stage, the algorithm scales the enhanced balanced semantic feature map to the size of each level feature map and enhances the original feature information with the original feature map, effectively addressing the imbalance between deep and shallow features. Consequently, the SIST detection model's detection performance is significantly improved, and the method is highly versatile. Experimental results show that the ABFP method can enhance traditional target detection methods, and is compatible with various models and feature fusion strategies.

STSIC: Swin-transformer-based scalable image coding for human and machine

Recently, as the visual data captured by smart terminal devices is mainly served for machine analysis, e.g., image classification, object detection, and sometimes also for human viewing in special scenarios, a new compression framework should be proposed to meet this demand. In this work, an image coding framework for machine analysis and human viewing simultaneously is proposed, termed STSIC. In order to leverage the relationship between the two tasks, STSIC is built upon a scalable structure, comprising a base layer for machine analysis and image preview at low quality, as well as an enhancement layer for high-quality image reconstruction. More specifically, features are extracted through the backbone and transmitted to the cloud side for tasks on the base layer. Different from the vast majority of existing works, we try to employ the Swin-Transformer as the backbone instead of a convolutional neural network (CNN). Additionally, due to the hierarchical structure of the Swin-Transformer, a feature fusion strategy is applied to generate a common feature that is more friendly to both human and machine tasks. The enhancement layer transfers the residuals between the original images and the reconstructed images from the base layer, thereby improving the quality of image reconstruction for human viewing. The experimental results show that STSIC outperforms relevant benchmarks on object detection, especially under low bitrates. As for image reconstruction, the advantage is weakened, but it still surpasses HEVC on the rate-distortion performance.

A deep learning-based approach for assessment of bridge condition through fusion of multi-type inspection data

Bridges typically undergo regular inspections to assess their structural conditions. However, relying solely on numerical data overlooks valuable information from other data types, reducing assessment reliability. Although data fusion is an effective solution, existing methods poorly handle defective data scenarios (sparse, imbalance, and loss). To address this issue, this study proposes a novel deep learning-based assessment model, the Bridge Information Fusion Network (BI-FusionNet). The features of the developed BI-FusionNet: (1) Feature extraction and processing layers can be compatibility with processing networks for various data types, extracting and unifying the key features of these data; (2) Innovative fusion technology combining SENet and the random fusion matrix, enabling deep fusion from data types to feature information. Appropriate extraction models mitigate sparse data effects via extracting critical features; the novel feature fusion strategy exploits cross-type information, resolving the imbalanced and missing data issues. The experimental results verified that the BI-FusionNet model achieved an accuracy of 0.9687 in assessing bridge conditions using normal dataset. In the effectiveness test, the model accuracy was 0.8377 by using the defective dataset, outperforming baseline methods. Therefore, the proposed BI-FusionNet can alleviate the issue of performance degradation from defective data and facilitates multi-type inspection data application in bridge condition evaluation.

Edge detection for 3D point clouds via locally max-angular gaps descriptor

Numerous point-cloud-based applications, including surface reconstruction and completion, greatly benefit from the utilization of edge features, which play a crucial role in structuring the target shape. Nevertheless, the current limitations of point cloud edge detection techniques in effectiveness and efficiency have motivated us to develop a more robust 3D edge detection method. In this paper, we introduce a simple yet effective descriptor for evaluating the edge level of 3D point clouds. This descriptor is defined as the variance of the max-angular gaps between the target point and its nearest neighboring points. Leveraging this descriptor, we further propose a multiscale feature fusion strategy that well detect the edge features for the target point cloud. Experimental results demonstrate the superiority of our method over other comparison methods in terms of effectiveness and stability in preserving point cloud edges. Moreover, we investigate the applications of our method in point cloud simplification and registration to provide additional insights into its potential use in some point-cloud-based downstream tasks.

FishRNFuseNET: development of heuristic-derived recurrent neural network with feature fusion strategy for fish species classification

FishRNFuseNET: development of heuristic-derived recurrent neural network with feature fusion strategy for fish species classification

DanceTrend: An Integration Framework of Video-Based Body Action Recognition and Color Space Features for Dance Popularity Prediction

Background: With the rise of user-generated content (UGC) platforms, we are witnessing an unprecedented surge in data. Among various content types, dance videos have emerged as a potent medium for artistic and emotional expression in the Web 2.0 era. Such videos have increasingly become a significant means for users to captivate audiences and amplify their online influence. Given this, predicting the popularity of dance videos on UGC platforms has drawn significant attention. Methods: This study postulates that body movement features play a pivotal role in determining the future popularity of dance videos. To test this hypothesis, we design a robust prediction framework DanceTrend to integrate the body movement features with color space information for dance popularity prediction. We utilize the jazz dance videos from the comprehensive AIST++ street dance dataset and segment each dance routine video into individual movements. AlphaPose was chosen as the human posture detection algorithm to help us obtain human motion features from the videos. Then, the ST-GCN (Spatial Temporal Graph Convolutional Network) is harnessed to train the movement classification models. These pre-trained ST-GCN models are applied to extract body movement features from our curated Bilibili dance video dataset. Alongside these body movement features, we integrate color space attributes and user metadata for the final dance popularity prediction task. Results: The experimental results endorse our initial hypothesis that the body movement features significantly influence the future popularity of dance videos. A comprehensive evaluation of various feature fusion strategies and diverse classifiers discern that a pre–post fusion hybrid strategy coupled with the XGBoost classifier yields the most optimal outcomes for our dataset.

Ship detection and identification in SDGSAT-1 glimmer images based on the glimmer YOLO model

ABSTRACTRemote sensing technology has been widely used for marine monitoring. However, due to the limitations of sensor technologies and data sources, effective monitoring of marine ships at night remains challenging. To address these challenges, our study developed SDGST, a high-resolution glimmer marine ship dataset from SDGSAT-1 satellite and proposed a ship detection and identification method based on the YOLOv5s model, the Glimmer YOLO model. Considering the characteristics of glimmer images, our model has made several effective improvements to the original YOLOv5s model. In particular, the improved model incorporates a new layer for detecting small targets and integrates the CA (Coordinate Attention) mechanism. To enhance the original feature fusion strategy, we introduced BiFPN (Bi-directional Feature Pyramid Network). We also adopted the EIOU Loss function and replaced the initially defined anchors with clustering results, thus improving detection performance. The mean Average Precision (mAP%) reaches 96.7%, which is a 5.1% improvement over the YOLOv5s model. Notably, it significantly improves the detection of small ships. This model demonstrates superior performance in ship detection under glimmer conditions compared to the original YOLOv5s model and other popular target detection models, and may serve as a valuable reference for achieving high-precision nighttime marine monitoring.

Semantic segmentation with cross convolution and multi-layer feature refinement

Multi-level features and contextual information have been proven effective in semantic segmentation. As a common solution, dilated convolution with multiple dilation rates is widely adopted by various computer vision tasks. However, since a large number of pixels are not involved in convolutional calculation, dilated convolution suffers from the information loss problem. Another important aspect for image segmentation is that most feature fusion methods combine different level features directly, which ignores the semantic gap between shallow layer features and deep layer features. In this work, we propose the multi-scale cross convolution to alleviate the information loss problem. Cross convolution conducts convolutional operations in horizontal and vertical direction with different kernels. By combining cross convolution with dilated convolutions using different convolution kernels, more pixels are engaged in convolutional operation to capture multi-scale features. To address the issue of semantic gap between multi-layer features, a feature fusion scheme is developed in which a dual attention mechanism is applied to conduct feature refinement in both spatial and channel dimensions. Comprehensive experiments are conducted to evaluate the proposed method on Cityscapes and ADE20K datasets. Experimental results demonstrate that the cross convolution and feature fusion method improve segmentation performance significantly and achieve competitive performance over state-of-the-art approaches.

Feature fusion strategy and improved GhostNet for accurate recognition of fish feeding behavior

In aquaculture, accurate detection of fish feeding intensity is a critical step for establishing an on-demand feeding system. In this paper, a novel fish feeding intensity detection method based on the fusion of multiple features (Mel spectrogram, STFT, and CQT feature map) and the LC-GhostNet lightweight network was proposed as an experimental object with Oplegnathus. First, a dataset of feeding sounds of Oplegnathus punctatus was built, in which there are four types: “strong”, “medium”, “weak” and “none”. Next, the Mel, STFT, and CQT feature maps of the feeding sound were extracted using the Librosa library, and then these feature maps were fused by feature image stitching. Lastly, the fused audio feature maps were fed to LC-GhostNet for further feature extraction and classification. Experimental results indicated that the accuracy of the fused feature maps was 97.941% when used as input, which was 4.053%, 7.207%, and 3.003% higher than that of the single feature Mel, STFT, and CQT, respectively. The method proposed in this paper is more lightweight and effective than the mainstream methods, and the accuracy has been improved. Based on this automatic and non-destructive method of obtaining fish feeding information, feeding decisions can be effectively optimized.

Energy-Efficient and High-Performance Ship Classification Strategy Based on Siamese Spiking Neural Network in Dual-Polarized SAR Images

Ship classification using the synthetic aperture radar (SAR) images has a significant role in remote sensing applications. Aiming at the problems of excessive model parameters numbers and high energy consumption in the traditional deep learning methods for the SAR ship classification, this paper provides an energy-efficient SAR ship classification paradigm that combines spiking neural networks (SNNs) with Siamese network architecture, for the first time in the field of SAR ship classification, which is called the Siam-SpikingShipCLSNet. It combines the advantage of SNNs in energy consumption and the advantage of the idea in performances that use the Siamese neuron network to fuse the features from dual-polarized SAR images. Additionally, we migrated the feature fusion strategy from CNN-based Siamese neural networks to the SNN domain and analyzed the effects of various spiking feature fusion methods on the Siamese SNN. Finally, an end-to-end error backpropagation optimization method based on the surrogate gradient has been adopted to train this model. Experimental results tested on the OpenSARShip2.0 dataset have demonstrated the correctness and effectiveness of the proposed SAR ship classification strategy, which has the advantages of the higher accuracy, fewer parameters and lower energy consumption compared with the mainstream deep learning method of the SAR ship classification.