Feature Representation Ability Research Articles

Enhancing local representation learning through global–local integration with functional connectivity for EEG-based emotion recognition

Emotion recognition based on electroencephalogram (EEG) signals is crucial in understanding human affective states. Current research has limitations in extracting local features. The representation capabilities of local features are limited, making it difficult to comprehensively capture emotional information. In this study, a novel approach is proposed to enhance local representation learning through global–local integration with functional connectivity for EEG-based emotion recognition. By leveraging the functional connectivity of brain regions, EEG signals are divided into global embeddings that represent comprehensive brain connectivity patterns throughout the entire process and local embeddings that reflect dynamic interactions within specific brain functional networks at particular moments. Firstly, a convolutional feature extraction branch based on the residual network is designed to extract local features from the global embedding. To further improve the representation ability and accuracy of local features, a multidimensional collaborative attention (MCA) module is introduced. Secondly, the local features and patch embedded local embeddings are integrated into the feature coupling module (FCM), which utilizes hierarchical connections and enhanced cross-attention to couple region-level features, thereby enhancing local representation learning. Experimental results on three public datasets show that compared with other methods, this method improves accuracy by 4.92% on the DEAP, by 1.11% on the SEED, and by 7.76% on the SEED-IV, demonstrating its superior performance in emotion recognition tasks.

Adversarial alignment and graph fusion via information bottleneck for multimodal emotion recognition in conversations

With the rapid development of social media and human–computer interaction, multimodal emotion recognition in conversations (MERC) tasks have begun to receive widespread research attention. The MERC task is to extract and fuse complementary semantic information from different modalities to classify the speaker’s emotion. However, the existing feature fusion methods usually directly map the features of other modalities into the same feature space for information fusion, which cannot eliminate the heterogeneity between different modalities and make the subsequent emotion class boundary learning more difficult. In addition, existing graph contrastive learning methods obtain consistent feature representations by maximizing mutual information between multiple views, which may lead to overfitting of the model. To tackle the above problem, we propose a novel Adversarial Alignment and Graph Fusion via Information Bottleneck for Multimodal Emotion Recognition in Conversations (AGF-IB) method. Firstly, we input video, audio, and text features into a multi-layer perceptron (MLP) to map them into separate feature spaces. Secondly, we build a generator and a discriminator for the three modal features, respectively, through adversarial representation to achieve information interaction between modalities and eliminate the heterogeneity among modalities. Thirdly, we introduce graph contrastive representation learning to capture intra-modal and inter-modal complementary semantic information and learn intra-class and inter-class boundary information of emotion categories. Furthermore, instead of maximizing the mutual information (MI) between multiple views, we use information bottleneck theory to minimize the MI between views. Specifically, we construct a graph structure for the three modal features respectively and perform contrastive representation learning on nodes with different emotions in the same modality and nodes with the same emotion in different modalities, to improve the feature representation ability of nodes. Finally, we use MLP to complete the emotional classification of the speaker. Extensive experiments show that AGF-IB can improve emotion recognition accuracy on IEMOCAP and MELD datasets. Furthermore, since AGF-IB is a general multimodal fusion and contrastive learning method, it can be applied to other multimodal tasks in a plug-and-play manner, e.g., humor detection.

A novel adaptive gramian angle field based intelligent fault diagnosis for motor rolling bearings

Abstract The Gramian Angle Field (GAF) can encode one-dimensional vibration signals into two-dimensional feature maps containing time correlation information. However, the feature representation ability of traditional GAF-encoded images is limited, making it challenging to further improve fault diagnosis accuracy. To address these issues, an AGAF-CNN intelligent fault diagnosis method is proposed in this paper. In addition to traditional GAF encoding, an adaptive adjustment parameter is introduced that automatically determines its value based on the dynamic distribution characteristics of the input data. This ensures that the resulting feature map after two-dimensional encoding contains optimal fault discriminant features. By leveraging CNN’s capability to process image data, the proposed method extracts deep-level feature representations from the vibration signal feature map and performs accurate diagnosis. The approach is validated using a rolling bearing dataset from Case Western Reserve University in the United States, and superior diagnostic accuracy is demonstrated compared to other vibration signal coding methods combined with CNN-based fault diagnosis methods. The theoretical and practical advantages of AGAF feature coding are explained comprehensively.

Interpretable domain adaptation transformer: a transfer learning method for fault diagnosis of rotating machinery

Domain adaptation-based transfer learning methods have been widely investigated in fault diagnosis of rotating machinery, but their basic convolution or recurrent structure is subject to poor global feature representation ability, which hinders the learning of domain-irrelevant modulation information. In addition, the “black box” nature of deep learning models limits their applications in high risk-sensitive scenarios. In this paper, an interpretable domain adaptation transformer (IDAT) is proposed for the transferable fault diagnosis of rotating machinery. First, a multi-layer domain adaptation transformer framework is proposed, which can capture the global information that is crucial for learning the modulation information of different domains, and meanwhile reduce the feature distribution discrepancy. Second, an ensemble attention weight is applied to enable the transfer learning framework to be interpretable, which is implemented by averaging the integral values of the multi-head attention maps along the key direction. In addition, the raw vibration signals are embedded as the input of the model, which provides an end-to-end fault diagnosis. The proposed IDAT is tested by various cross-condition and cross-machine bearing fault diagnosis tasks, and results confirm the advantages of the method.

Spatiotemporal Inconsistency Learning and Interactive Fusion for Deepfake Video Detection

With the rise of the metaverse, the rapid advancement of Deepfakes technology has become closely intertwined. Within the metaverse, individuals exist in digital form and engage in interactions, transactions, and communications through virtual avatars. However, the development of Deepfakes technology has led to the proliferation of forged information disseminated under the guise of users’ virtual identities, posing significant security risks to the metaverse. Hence, there is an urgent need to research and develop more robust methods for detecting deep forgeries to address these challenges. This paper explores deepfake video detection by leveraging the spatiotemporal inconsistencies generated by deepfake generation techniques, and thereby proposing the interactive spatioTemporal inconsistency learning and interactive fusion (ST-ILIF) detection method, which consists of phase-aware and sequence streams. The spatial inconsistencies exhibited in frames of deepfake videos are primarily attributed to variations in the structural information contained within the phase component of the Fourier domain. To mitigate the issue of overfitting the content information, a phase-aware stream is introduced to learn the spatial inconsistencies from the phase-based reconstructed frames. Additionally, considering that deepfake videos are generated frame-by-frame and lack temporal consistency between frames, a sequence stream is proposed to extract temporal inconsistency features from the spatiotemporal difference information between consecutive frames. Finally, through feature interaction and fusion of the two streams, the representation ability of intermediate and classification features is further enhanced. The proposed method, which was evaluated on four mainstream datasets, outperformed most existing methods, and extensive experimental results demonstrated its effectiveness in identifying deepfake videos. Our source code is available at https://github.com/qff98/Deepfake-Video-Detection

DeFRCN-MAM: DeFRCN and multi-scale attention mechanism-based industrial defect detection method

ABSTRACT With the technology development, industrial defect detection based on deep learning has attracted extensive attention in the academic community. Different from general visual objects, industrial defects have the characteristics of small sample, weak visibility and irregular shape, which hinder the application of related studies. According to these problems, a few-shot object detection (FSOD) method based on Decoupled Faster R-CNN (DeFRCN) is proposed in this paper. Firstly, it includes fine-tuning processing, because of the small sample characteristics. To adapt to the invisible characteristics of defects, we introduce the Feature Pyramid Network (FPN) and Residual Attention Module (RAM) into DeFRCN, which can enhance the capture ability of multi-scale features and feature association information. Furthermore, the feature representation ability is strengthened by parallel connecting of two channels, consisting of R-CNN head, box classifier and box regression models. Finally, it is completed that the pre-training, fine-tuning and testing of the proposed network, with DAGM 2007 and NEU-DET public industrial defect datasets as the base class and flange shaft defect data collected in the laboratory as the new class. To verify the effectiveness of the proposed one, we compare them with other classical FSOD methods. The superiority of the proposed method is obvious.

Multi-scale recurrent attention gated fusion network for single image dehazing

The purpose of single image dehazing is to eliminate the bad influence of haze on images, so as to maintain more scene information of images. In recent years, the convolutional neural networks (CNN) have made significant contributions to single image dehazing. However, the visual quality of dehazed images still needs to be further improved. In view of the problems of single-scale shallow image feature extraction and the insufficient use of intermediate layer features in existing dehazing networks, we propose an end-to-end Multi-scale Recurrent Attention Gated Fusion Network (MRAGFN) to address the image dehazing task. We cascade three Dual Attention Fusion (DAF) modules to progressively form three haze-relevant features map, meanwhile, we adopt downsampling operation on the input to produce global feature map, which are used to weight the three feature maps to compensate for the missing of single-scale feature information. We present Feature Enhancement Module (FEM) to enhance the feature representation ability of these weighted feature maps. We design Recurrent Attention Gated Fusion (RAGF) module by adding attention mechanism and gating mechanism to gradually obtain more refined features based on these weighted features while eliminating redundant features. Experimental results on different hazy images demonstrate that the proposed dehazing network can restore the haze-free images and perform better than the state-of-the-art dehazing networks in terms of the objective indicators (such as PSNR, SSIM)and the subjective visual quality.

Text-image semantic relevance identification for aspect-based multimodal sentiment analysis

Aspect-based multimodal sentiment analysis (ABMSA) is an emerging task in the research of multimodal sentiment analysis, which aims to identify the sentiment of each aspect mentioned in multimodal sample. Although recent research on ABMSA has achieved some success, most existing models only adopt attention mechanism to interact aspect with text and image respectively and obtain sentiment output through multimodal concatenation, they often neglect to consider that some samples may not have semantic relevance between text and image. In this article, we propose a Text-Image Semantic Relevance Identification (TISRI) model for ABMSA to address the problem. Specifically, we introduce a multimodal feature relevance identification module to calculate the semantic similarity between text and image, and then construct an image gate to dynamically control the input image information. On this basis, an image auxiliary information is provided to enhance the semantic expression ability of visual feature representation to generate more intuitive image representation. Furthermore, we employ attention mechanism during multimodal feature fusion to obtain the text-aware image representation through text-image interaction to prevent irrelevant image information interfering our model. Experiments demonstrate that TISRI achieves competitive results on two ABMSA Twitter datasets, and then validate the effectiveness of our methods.

Deep Reinforcement Learning: A Survey.

Deep reinforcement learning (DRL) integrates the feature representation ability of deep learning with the decision-making ability of reinforcement learning so that it can achieve powerful end-to-end learning control capabilities. In the past decade, DRL has made substantial advances in many tasks that require perceiving high-dimensional input and making optimal or near-optimal decisions. However, there are still many challenging problems in the theory and applications of DRL, especially in learning control tasks with limited samples, sparse rewards, and multiple agents. Researchers have proposed various solutions and new theories to solve these problems and promote the development of DRL. In addition, deep learning has stimulated the further development of many subfields of reinforcement learning, such as hierarchical reinforcement learning (HRL), multiagent reinforcement learning, and imitation learning. This article gives a comprehensive overview of the fundamental theories, key algorithms, and primary research domains of DRL. In addition to value-based and policy-based DRL algorithms, the advances in maximum entropy-based DRL are summarized. The future research topics of DRL are also analyzed and discussed.

Bibliometric and visualized analysis of DME from 2012 to 2022.

Diabetic macular edema (DME) is the main cause of irreversible vision loss in patients with diabetes mellitus (DM), resulting in a certain burden to patients and society. With the increasing incidence of DME, more and more researchers are focusing on it. The papers related to DME between 2012 and 2022 from the Web of Science core Collection were searched in this study. Based on CiteSpace and VOS viewer, these publications were analyzed in terms of spatiotemporal distribution, author distribution, subject classification, topic distribution, and citations. A total of 5165 publications on DME were included. The results showed that the research on DME is on a steady growth trend. The country with the highest number of published documents was the US. Wong Tien Yin from Tsinghua University was the author with the most published articles. The journal of Retina, the Journal of Retinal and Vitreous Diseases had a large number of publications. The article "Mechanisms of macular edema: Beyond the surface" was the highly cited literature and "Aflibercept, bevacizumab, or ranibizumab for diabetic macular edema" had the highest co-citation frequency. The treatment, diagnosis, pathogenesis, as well as etiology and epidemiological investigation of DME, have been the current research direction. Deep learning has been widely used in the medical field for its strong feature representation ability. The study revealed the important authoritative literature, journals, institutions, scholars, countries, research hotspots, and development trends in in the field of DME. This indicates that communication and cooperation between disciplines, universities, and countries are crucial. It can advance research in DME and even ophthalmology.

CLIP-Guided Federated Learning on Heterogeneity and Long-Tailed Data

Federated learning (FL) provides a decentralized machine learning paradigm where a server collaborates with a group of clients to learn a global model without accessing the clients' data. User heterogeneity is a significant challenge for FL, which together with the class-distribution imbalance further enhances the difficulty of FL. Great progress has been made in large vision-language models, such as Contrastive Language-Image Pre-training (CLIP), which paves a new way for image classification and object recognition. Inspired by the success of CLIP on few-shot and zero-shot learning, we use CLIP to optimize the federated learning between server and client models under its vision-language supervision. It is promising to mitigate the user heterogeneity and class-distribution balance due to the powerful cross-modality representation and rich open-vocabulary prior knowledge. In this paper, we propose the CLIP-guided FL (CLIP2FL) method on heterogeneous and long-tailed data. In CLIP2FL, the knowledge of the off-the-shelf CLIP model is transferred to the client-server models, and a bridge is built between the client and server. Specifically, for client-side learning, knowledge distillation is conducted between client models and CLIP to improve the ability of client-side feature representation. For server-side learning, in order to mitigate the heterogeneity and class-distribution imbalance, we generate federated features to retrain the server model. A prototype contrastive learning with the supervision of the text encoder of CLIP is introduced to generate federated features depending on the client-side gradients, and they are used to retrain a balanced server classifier. Extensive experimental results on several benchmarks demonstrate that CLIP2FL achieves impressive performance and effectively deals with data heterogeneity and long-tail distribution. The code is available at https://github.com/shijiangming1/CLIP2FL.

Efficient Lightweight Image Denoising with Triple Attention Transformer

Transformer has shown outstanding performance on image denoising, but the existing Transformer methods for image denoising are with large model sizes and high computational complexity, which is unfriendly to resource-constrained devices. In this paper, we propose a Lightweight Image Denoising Transformer method (LIDFormer) based on Triple Multi-Dconv Head Transposed Attention (TMDTA) to boost computational efficiency. LIDFormer first implements Discrete Wavelet Transform (DWT), which transforms the input image into a low-frequency space, greatly reducing the computational complexity of image denoising. However, the low-frequency image lacks fine-feature information, which degrades the denoising performance. To handle this problem, we introduce the Complementary Periodic Feature Reusing (CPFR) scheme for aggregating the shallow-layer features and the deep-layer features. Furthermore, TMDTA is proposed to integrate global context along three dimensions, thereby enhancing the ability of global feature representation. Note that our method can be applied as a pipeline for both convolutional neural networks and Transformers. Extensive experiments on several benchmarks demonstrate that the proposed LIDFormer achieves a better trade-off between high performance and low computational complexity on real-world image denoising tasks.

Real-Time Detection of Drill Pipe Joints Using Improved YOLOv5x Model Applied to Drilling Operation Images

It is widely known that pipe assembly and disassembly still lacks automation. To enhance drilling efficiency, we propose an autonomous detection and positioning model based on improved YOLOv5x for drill pipe joints. Firstly, we use Activate or Not (ACON) activation function and Convolution Block Attention Module (CBAM) to enhance feature extraction and representation ability. Then, the loss function is changed from Complete IoU (CIOU) loss to Scale-Invariant IoU (SIOU) loss, which increases the accuracy of the bounding box regression. Finally, the predict heads are tailored to be more effective in detecting tiny targets. Following network training, the final drill pipe joint detection model, based on the improved YOLOv5x, achieved an average accuracy of 99.10%, a 3.5% improvement over the base YOLOv5x algorithm. The proposed method effectively promotes the intelligence of oil drilling equipment by quickly and accurately detecting drill pipe joints, consequently enhancing drilling operation efficiency.

Hyperspectral Image Classification Based on 3D–2D Hybrid Convolution and Graph Attention Mechanism

Convolutional neural networks and graph convolutional neural networks are two classical deep learning models that have been widely used in hyperspectral image classification tasks with remarkable achievements. However, hyperspectral image classification models based on graph convolutional neural networks using only shallow spectral or spatial features are insufficient to provide reliable similarity measures for constructing graph structures, limiting their classification performance. To address this problem, we propose a new end-to-end hyperspectral image classification model combining 3D–2D hybrid convolution and a graph attention mechanism (3D–2D-GAT). The model utilizes the collaborative work of hybrid convolutional feature extraction module and GAT module to improve classification accuracy. First, a 3D–2D hybrid convolutional network is constructed and used to quickly extract the discriminant deep spatial-spectral features of various ground objects in hyperspectral image. Then, the graph is built based on deep spatial-spectral features to enhance the feature representation ability. Finally, a network of graph attention mechanism is adopted to learn long-range spatial relationship and distinguish the intra-class variation and inter-class similarity among different samples. The experimental results on three datasets, Indian Pine, the University of Pavia and Salinas Valley show that the proposed method can achieve higher classification accuracy compared with other advanced methods.

A Parallel Convolutional Network Based on Spiking Neural Systems.

Deep convolutional neural networks have shown advanced performance in accurately segmenting images. In this paper, an SNP-like convolutional neuron structure is introduced, abstracted from the nonlinear mechanism in nonlinear spiking neural P (NSNP) systems. Then, a U-shaped convolutional neural network named SNP-like parallel-convolutional network, or SPC-Net, is constructed for segmentation tasks. The dual-convolution concatenate (DCC) and dual-convolution addition (DCA) network blocks are designed, respectively, in the encoder and decoder stages. The two blocks employ parallel convolution with different kernel sizes to improve feature representation ability and make full use of spatial detail information. Meanwhile, different feature fusion strategies are used to fuse their features to achieve feature complementarity and augmentation. Furthermore, a dual-scale pooling (DSP) module in the bottleneck is designed to improve the feature extraction capability, which can extract multi-scale contextual information and reduce information loss while extracting salient features. The SPC-Net is applied in medical image segmentation tasks and is compared with several recent segmentation methods on the GlaS and CRAG datasets. The proposed SPC-Net achieves 90.77% DICE coefficient, 83.76% IoU score and 83.93% F1 score, 86.33% ObjDice coefficient, 135.60 Obj-Hausdorff distance, respectively. The experimental results show that the proposed model can achieve good segmentation performance.

CD-MAE: Contrastive Dual-Masked Autoencoder Pre-Training Model for PCB CT Image Element Segmentation

Element detection is an important step in the process of the non-destructive testing of printed circuit boards (PCB) based on computed tomography (CT). Compared with the traditional manual detection method, the image semantic segmentation method based on deep learning greatly improves efficiency and accuracy. However, semantic segmentation models often require a large amount of data for supervised training to generalize better model performance. Unlike natural images, the PCB CT image annotation task is more time-consuming and laborious than the semantic segmentation task. In order to reduce the cost of labeling and improve the ability of the model to utilize unlabeled data, unsupervised pre-training is a very reasonable and necessary choice. The masked image reconstruction model represented by a masked autoencoder is pre-trained on the unlabeled data, learning a strong feature representation ability by recovering the masked image, and shows a good generalization ability in various downstream tasks. In the PCB CT image element segmentation task, considering the characteristics of the image, it is necessary to use a model with strong feature robustness in the pre-training stage to realize the representation learning on a large number of unlabeled PCB CT images. Based on the above purposes, we proposed a contrastive dual-masked autoencoder (CD-MAE) pre-training model, which can learn more robust feature representation on unlabeled PCB CT images. Our experiments show that the CD-MAE outperforms the baseline model and fully supervised models in the PCB CT element segmentation task.

Multi-axis interactive multidimensional attention network for vehicle re-identification

Learning fine-grained discriminative information is essential to address the challenges of small inter-class differences and large intra-class differences in vehicle re-identification (Re-ID). Attentional mechanism is often used to capture important global information in images rather than fine-grained discriminative information. Studies have shown that the multi-axis interaction of information can enhance the feature representation ability of networks. This paper explores how to use the multi-axis interaction of information to facilitate more effective learning of attention and how to capture important detailed information in local regions. We propose a multi-axis interactive multidimensional attention network (MIMA-Net) for vehicle Re-ID. The network allows information to interact on multiple axes and calibrates the weight distribution of features from multiple dimensions to learn subtle discriminative information in vehicle parts/regions. The window-channel attention module (W-CAM) in MIMA-Net facilitates the learning of channel attention by interacting first across locations and then across channels, while the channel group-spatial attention module (CG-SAM) facilitates the learning of spatial attention by interacting first across channels and then across locations. These two modules perform window partitioning in a priori manner and channel semantic aggregation in an adaptive manner to learn discriminative semantic features in parts, respectively. These two approaches complement each other to strengthen the feature representation ability of MIMA-Net. Extensive experiments on three large public datasets, VeRi-776, VehicleID, and VERI-Wild, verify the effectiveness of our MIMA-Net and show that our method achieves state-of-the-art performance.

DI-VTR: Dual inter-modal interaction model for video-text retrieval

Video-text retrieval is a challenging task for multimodal information processing due to the semantic gap between different modalities. However, most existing methods do not fully mine the intra-modal interactions, as with the temporal correlation of video frames, which results in poor matching performance. Additionally, the imbalanced semantic information between videos and texts also leads to difficulty in the alignment of the two modalities. To this end, we propose a dual inter-modal interaction network for video-text retrieval, i.e., DI-VTR. To learn the intra-modal interaction of video frames, we design a contextual-related video encoder to obtain more fine-grained content-oriented video representations. We also propose a dual inter-modal interaction module to accomplish accurate multilingual alignment between the video and text modalities by introducing multilingual text to improve the representation ability of text semantic features. Extensive experimental results on commonly-used video-text retrieval datasets, including MSR-VTT, MSVD and VATEX, show that the proposed method achieves significantly improved performance compared with state-of-the-art methods.

A Practical Deep Learning Architecture for Large-Area Solid Wastes Monitoring Based on UAV Imagery

The development of global urbanization has brought about a significant amount of solid waste. These untreated wastes may be dumped in any corner, causing serious pollution to the environment. Thus, it is necessary to accurately obtain their distribution locations and detailed edge information. In this study, a practical deep learning network for recognizing solid waste piles over extensive areas using unmanned aerial vehicle (UAV) imagery has been proposed and verified. Firstly, a high-resolution dataset serving to solid waste detection was created based on UAV aerial data. Then, a dual-branch solid waste semantic segmentation model was constructed to address the characteristics of the integration of solid waste distribution with the environment and the irregular edge morphology. The Context feature branch is responsible for extracting high-level semantic features, while the Spatial feature branch is designed to capture fine-grained spatial details. After information fusion, the model obtained more comprehensive feature representation and segmentation ability. The effectiveness of the improvement was verified through ablation experiments and compared with 13 commonly used semantic segmentation models, demonstrating the advantages of the method in solid waste segmentation tasks, with an overall accuracy of over 94%, and a recall rate of 88.6%—much better than the best performing baselines. Finally, a spatial distribution map of solid waste over Jiaxing district, China was generated by the model inference, which assisted the environmental protection department in completing environmental management. The proposed method provides a feasible approach for the accurately monitoring of solid waste, so as to provide policy support for environmental protection.

S3-Net: A Self-Supervised Dual-Stream Network for Radiology Report Generation.

Intelligent medicine is eager to automatically generate radiology reports to ease the tedious work of radiologists. Previous researches mainly focused on the text generation with encoder-decoder structure, while CNN networks for visual features ignored the long-range dependencies correlated with textual information. Besides, few studies exploit cross-modal mappings to promote radiology report generation. To alleviate the above problems, we propose a novel end-to-end radiology report generation model dubbed Self-Supervised dual-Stream Network (S3-Net). Specifically, a Dual-Stream Visual Feature Extractor (DSVFE) composed of ResNet and SwinTransformer is proposed to capture more abundant and effective visual features, where the former focuses on local response and the latter explores long-range dependencies. Then, we introduced the Fusion Alignment Module (FAM) to fuse the dual-stream visual features and facilitate alignment between visual features and text features. Furthermore, the Self-Supervised Learning with Mask(SSLM) is introduced to further enhance the visual feature representation ability. Experimental results on two mainstream radiology reporting datasets (IU X-ray and MIMIC-CXR) show that our proposed approach outperforms previous models in terms of language generation metrics.