Multi-grained Features Research Articles

Multi-Scale Adaptive Skeleton Transformer for action recognition

Transformer has demonstrated remarkable performance in various computer vision tasks. However, its potential is not fully explored in skeleton-based action recognition. On one hand, existing methods primarily utilize fixed function or pre-learned matrix to encode position information, while overlooking the sample-specific position information. On the other hand, these approaches focus on single-scale spatial relationships, while neglecting the discriminative fine-grained and coarse-grained spatial features. To address these issues, we propose a Multi-Scale Adaptive Skeleton Transformer (MSAST), including Adaptive Skeleton Position Encoding Module (ASPEM), Multi-Scale Embedding Module (MSEM), and Adaptive Relative Location Module (ARLM). ASPEM decouples spatial–temporal information in the position encoding procedure, which acquires inherent dependencies of skeleton sequences. ASPEM is also designed to be dependent on input tokens, which can learn sample-specific position information. The MSEM employs multi-scale pooling to generate multi-scale tokens that contain multi-grained features. Then, the spatial transformer captures multi-scale relations to address the subtle differences between various actions. Another contribution of this paper is that ARLM is presented to mine suitable location information for better recognition performance. Extensive experiments conducted on three benchmark datasets demonstrate that the proposed model achieves Top-1 accuracy of 94.9%/97.5% on NTU-60 C-Sub/C-View, 88.7%/91.6% on NTU-120 X-Sub/X-Set and 97.4% on NW-UCLA, respectively.

Granular3D: Delving into multi-granularity 3D scene graph prediction

This paper addresses the significant challenges in 3D Semantic Scene Graph (3DSSG) prediction, essential for understanding complex 3D environments. Traditional approaches, primarily using PointNet and Graph Convolutional Networks, struggle with effectively extracting multi-grained features from intricate 3D scenes, largely due to a focus on global scene processing and single-scale feature extraction. To overcome these limitations, we introduce Granular3D, a novel approach that shifts the focus towards multi-granularity analysis by predicting relation triplets from specific sub-scenes. One key is the Adaptive Instance Enveloping Method (AIEM), which establishes an approximate envelope structure around irregular instances, providing shape-adaptive local point cloud sampling, thereby comprehensively covering the contextual environments of instances. Moreover, Granular3D incorporates a Hierarchical Dual-Stage Network (HDSN), which differentiates and processes features of instances and their pairs at varying scales, leading to a targeted prediction of instance categories and their relationships. To advance the perception of sub-scene in HDSN, we design a Gather Point Transformer structure (GaPT) that enables the combinatorial interaction of local information from multiple point cloud sets, achieving a more comprehensive local contextual feature extraction. Extensive evaluations on the challenging 3DSSG benchmark demonstrate that our methods provide substantial improvements, establishing a new state-of-the-art in 3DSSG prediction, boosting the top-50 triplet accuracy by ＋2.8%.

Fault diagnosis of helicopter tail-drive system using a multi-grained hierarchical message graph convolutional networks

ABSTRACT The fault diagnosis of the tail-drive of helicopter is a crucial task for helicopter system operation and maintenance. Recently, graph convolution network (GCN) has been the focus in fault diagnosis for its powerful representational ability in relationship mining. However, with the difficulty of obtaining node and edge information in the high-order domain, the stable performance of the long-range message-passing process of the deep GCN is unknown limits the application of GCN in fault diagnosis. To address these issues, a multi-grained hierarchical message graph convolutional network (MHGCN) is proposed to diagnose faults of helicopter tail-drive system. First, time-frequency characteristics of the original vibration signals are extracted to construct the graph nodes. The original graph nodes are aggregated by Louvain community detection, which can effectively learn the multi-grained features. Then, the hierarchical graph is introduced to learn the features of high-order neighbourhoods. Finally, a particular message-passing method is used to encode long-range information spanning the graph structure and realise accurate classification. Experiments on a test rig of helicopter tail-drive system are performed to verify the efficacy of the proposed method.

Multi-Grained feature aggregation based on Transformer for unsupervised person re-identification

Multi-Grained feature aggregation based on Transformer for unsupervised person re-identification

Enhancing identification for person search with multi-scale multi-grained representation learning

Person Search aims to simultaneously address Person Detection and Person Re-ID. There are various challenges in person search such as significant scale variations, occlusions, and partial instances. In this paper, we propose a Multi-Scale Multi-Grained (MSMG) sequential network for end-to-end person search, intended to alleviate these issues. To generate re-id representations robust to scale changes, MSMG leverages multi-scale RoI features and aggregates them with a proposed Multi-Scale feature Aggregation Encoder (MSAE). In this way, the aggregated multi-scale re-id features are enriched with more semantic information and detailed information, thereby being more discriminative for identification. Moreover, to produce re-id representations more robust to occlusions and partial instances, MSMG introduces a Multi-Grained feature Learning Decoder (MGLD) focused on multi-grained feature learning. MGLD adaptively decodes multi-grained re-id representations with more accurate semantic information through a regional deformable cross-attention module. Finally, the multi-scale multi-grained re-id representation substantially improves the identification accuracy under challenging cases. Through comprehensive experiments, we demonstrate that our method achieves state-of-the-art performance on two benchmark datasets. On the challenging PRW benchmark, MSMG obtains the best-reported results with a mean average precision (mAP) score of 61.3%.

MLMG-SGG: Multi-Label Scene Graph Generation with Multi-Grained Features.

As an important and challenging problem in computer vision, scene graph generation (SGG) aims to find out the underlying semantic relationships among objects from a given image for scene understanding. Usually, prevalent SGG approaches adopt a learning pipeline with the assumption that there exists only a single relationship for a particular object pair. Considering the common phenomenon that a pair of objects can be attached by multiple relationships, we propose a multi-label scene graph generation pipeline with multi-grained features (MLMG-SGG), which formulates the relationship detection as a multi-label classification problem during training while generating multigraphs at inference time. In order to better model the fine-grained relationships, the proposed pipeline encodes the feature representation of SGG on different spatial scales by a specially designed Multi-Grained Module (MGM), resulting in the multi-grained (i.e., object-level and region-level) features of objects. Experimental results over the benchmark dataset demonstrate the significant performance gain of the proposed pipeline used as a plug-in for the state-of-the-art methods.

MJ-GAN: Generative Adversarial Network with Multi-Grained Feature Extraction and Joint Attention Fusion for Infrared and Visible Image Fusion.

The challenging issues in infrared and visible image fusion (IVIF) are extracting and fusing as much useful information as possible contained in the source images, namely, the rich textures in visible images and the significant contrast in infrared images. Existing fusion methods cannot address this problem well due to the handcrafted fusion operations and the extraction of features only from a single scale. In this work, we solve the problems of insufficient information extraction and fusion from another perspective to overcome the difficulties in lacking textures and unhighlighted targets in fused images. We propose a multi-scale feature extraction (MFE) and joint attention fusion (JAF) based end-to-end method using a generative adversarial network (MJ-GAN) framework for the aim of IVIF. The MFE modules are embedded in the two-stream structure-based generator in a densely connected manner to comprehensively extract multi-grained deep features from the source image pairs and reuse them during reconstruction. Moreover, an improved self-attention structure is introduced into the MFEs to enhance the pertinence among multi-grained features. The merging procedure for salient and important features is conducted via the JAF network in a feature recalibration manner, which also produces the fused image in a reasonable manner. Eventually, we can reconstruct a primary fused image with the major infrared radiometric information and a small amount of visible texture information via a single decoder network. The dual discriminator with strong discriminative power can add more texture and contrast information to the final fused image. Extensive experiments on four publicly available datasets show that the proposed method ultimately achieves phenomenal performance in both visual quality and quantitative assessment compared with nine leading algorithms.

Infrared and visible image fusion based on double fluid pyramids and multi-scale gradient residual block

Infrared and visible image fusion can capture multi-modal features and generate an informative fused image. However, most existing encoder and decoder-based fusion methods cannot fully utilize the output features of each layer in the deep learning network. In most cases, there are no elaborate modules to fully extract the detail information of multi-scale feature maps. To this end, this work proposes a novel encoder and decoder-based infrared and visible image fusion network named PG-Fusion. In feature extraction part, the double fluid pyramids are designed to integrate and preserve features of each layer of the encoder in two independent data flows. Multi-layer structure and injected attention mechanisms bridge the gaps between low-level features and high-level features and promote the interaction of multi-grained features. Specifically, a multi-scale gradient residual block is devised to preserve enough fine-grained textures and details from source images, which can facilitate the extraction of discriminative information and boost the description ability of the network. In addition, sliding window comparison strategy based intensity loss function and gradient loss function are utilized to impel our network towards a final optimal solution. Extensive qualitative and quantitative experiment results illustrate that PG-Fusion achieves better fusion performance compared with eleven typical algorithms.

A Lightweight Sentiment Analysis Framework for a Micro-Intelligent Terminal

Sentiment analysis aims to mine polarity features in the text, which can empower intelligent terminals to recognize opinions and further enhance interaction capabilities with customers. Considerable progress has been made using recurrent neural networks or pre-trained models to learn semantic representations. However, recently published models with complex structures require increasing computational resources to reach state-of-the-art (SOTA) performance. It is still a significant challenge to deploy these models to run on micro-intelligent terminals with limited computing power and memory. This paper proposes a lightweight and efficient framework based on hybrid multi-grained embedding on sentiment analysis (MC-GGRU). The gated recurrent unit model is designed to incorporate a global attention structure that allows contextual representations to be learned from unstructured text using word tokens. In addition, a multi-grained feature layer can further enrich sentence representation features with implicit semantics from characters. Through hybrid multi-grained representation, MC-GGRU achieves high inference performance with a shallow structure. The experimental results of five public datasets show that our method achieves SOTA for sentiment classification with a trade-off between accuracy and speed.

CAFE and SOUP: Toward Adaptive VDI Workload Prediction

For Virtual Desktop Infrastructure (VDI) system, effective resource management is rather important where turning off spare virtual machines would help save running cost while maintaining sufficient virtual machines is essential to secure satisfactory user experience. Current VDI resource management strategy works in a passive manner by either reactively driving available capacity based on user demands or following manually configured schedules, which may lead to unnecessary running costs or unsatisfactory user experience. In this article, we propose a first attempt toward proactive VDI resource management, where two adaptive learning approaches for VDI workload prediction are proposed by learning from multi-grained historical features. For non-persistent desktop pool, based on the aggregation session count of pool-sharing users, the CAFE approach induces a pool-level workload predictive model by utilizing coarse-to-fine historical features extracted from aggregation workload data. For persistent desktop pool, based on the session connection status of individual users within the same pool, the SOUP approach induces user-level workload predictive model by incorporating encoded multi-grained features extracted from the logon behavior of individual users into an aggregation pool-level model. Extensive experiments on datasets of real VDI customers and electricity load evidently verify the effectiveness of the proposed adaptive approaches for VDI workload prediction as well as other workload prediction tasks.

Deep multi-sequence multi-grained cascade forest for tobacco drying condition identification

The term, “drying condition” refers to the actual dehydration capacity of a rotary dryer in the tobacco drying process which is directly related to the drying effect. However, identifying different drying conditions relies heavily on the judgment of field engineers who have rich domain knowledge and practical experiences. In this study, we proposed an improved multi-sequence multi-grained cascade forest model, MSgcForest, to classify and identify different drying conditions. An improved multi-sequence multi-grained feature scanning mechanism is added to perform spacial and sequential feature extraction from raw production-related data, which transforms the input features into high-dimensional feature vectors and increases the discriminative power of the drying condition features. Comparison with existing models indicates that the proposed MSgcForest outperforms the other alternatives even for small-scale training data. In particular, this method successfully transforms the fuzzy artificial judgment of the drying condition into a data-driven identification with high precision, which provides a promising prospect for identifying working conditions in industrial processes.

MGFN: A Multi-Granularity Fusion Convolutional Neural Network for Remote Sensing Scene Classification

Convolutional neural networks (CNNs) have been successfully used in remote sensing scene classification and identification due to their ability to capture deep spatial feature representations. However, the performance of deep models inevitably encounters a bottleneck when multimodality-dominated scene classification rather than single-modality-dominated scene classification is performed, due to the high similarity among different categories. In this study, we propose a novel multi-granularity fusion convolutional neural network (MGFN) to automatically capture the latent ontological features of remote sensing images. We firstly design a multigranularity module that can progressively crop input images to learn multigrained features, which can describe images to different degrees. Based on a comparison of different granularities, we then design a maxout-based module to learn the corresponding Gaussian covariance matrices of different granularities, which can extract second-order features to express the latent ontological essence of inputs and select the most distinguished inputs. We thirdly provide an adaptive fusion module to fuse all features via normalization to combine features of different degrees using the adaptive fused module. Finally, an SVM classifier is used to classify the fused matrix of every input image. Extensive experimentation and evaluations, particularly for multimodality-dominated scenes, demonstrate that the proposed network can achieve promising results for public remote sensing datasets.

Selective Domain-Invariant Feature Alignment Network for Face Anti-Spoofing

One primary challenge in face anti-spoofing refers to suffering a sharp performance drop in cross-domain scenes, where training and testing images are collected from different datasets. Recent methods have achieved promising results by aligning the features of all images among the available source domains. However, due to significant distribution discrepancies among non-face regions of all images, it is challenging to capture domain-invariant features for these regions. In this paper, we propose a novel Selective Domain-invariant Feature Alignment Network (SDFANet) for cross-domain face anti-spoofing, which aims to seek common feature representations by fully exploring the generalization of different regions of images. Different from previous works that align the whole features directly, the proposed SDFANet leverages multiple domain discriminators with the same architecture to balance the generalization of different regions of the all images. Specifically, we firstly design a multi-grained feature alignment network composed of a local-region and global-image alignment subnetworks to learn more generalized feature space for real faces. Besides, the domain adapter module, which aims to alleviate the large domain discrepancy with the help of the domain attention strategy, is adopted to facilitate the learning of our multi-grained feature alignment network. In addition, a multi-scale attention fusion module is designed in our feature generator to refine the different levels of features effectively. Experimental results show that the proposed SDFANet can greatly improve the generalization ability of face anti-spoofing, and that is superior to the existing methods.

WiGAN: A WiFi Based Gesture Recognition System with GANs.

In recent years, a series of research experiments have been conducted on WiFi-based gesture recognition. However, current recognition systems are still facing the challenge of small samples and environmental dependence. To deal with the problem of performance degradation caused by these factors, we propose a WiFi-based gesture recognition system, WiGAN, which uses Generative Adversarial Network (GAN) to extract and generate gesture features. With GAN, WiGAN expands the data capacity to reduce time cost and increase sample diversity. The proposed system extracts and fuses multiple convolutional layer feature maps as gesture features before gesture recognition. After fusing features, Support Vector Machine (SVM) is exploited for human activity classification because of its accuracy and convenience. The key insight of WiGAN is to generate samples and merge multi-grained feature maps in our designed GAN, which not only enhances the data but also allows the neural network to select different grained features for gesture recognition. According to the result of experiments conducted on two existing datasets, the average recognition accuracy of WiGAN reaches 98% and 95.6%, respectively, outperforming the existing system. Moreover, the recognition accuracy under different experimental environments and different users shows the robustness of WiGAN.

Multi-Granularity Canonical Appearance Pooling for Remote Sensing Scene Classification.

Recognising remote sensing scene images remains challenging due to large visual-semantic discrepancies. These mainly arise due to the lack of detailed annotations that can be employed to align pixel-level representations with high-level semantic labels. As the tagging process is labour-intensive and subjective, we hereby propose a novel Multi-Granularity Canonical Appearance Pooling (MG-CAP) to automatically capture the latent ontological structure of remote sensing datasets. We design a granular framework that allows progressively cropping the input image to learn multi-grained features. For each specific granularity, we discover the canonical appearance from a set of pre-defined transformations and learn the corresponding CNN features through a maxout-based Siamese style architecture. Then, we replace the standard CNN features with Gaussian covariance matrices and adopt the proper matrix normalisations for improving the discriminative power of features. Besides, we provide a stable solution for training the eigenvalue-decomposition function (EIG) in a GPU and demonstrate the corresponding back-propagation using matrix calculus. Extensive experiments have shown that our framework can achieve promising results in public remote sensing scene datasets.

CAFE: Adaptive VDI Workload Prediction with Multi-Grained Features

Virtual desktop infrastructure (VDI) is a virtualization technology that hosts desktop operating system on centralized server in a data center of private or public cloud. Effective resource management is of crucial importance for VDI customers, where maintaining sufficient virtual machines helps guarantee satisfactory user experience while turning off spare virtual machines helps save running cost. Generally, existing techniques work in passive manner by either driving available capacity reactively or configuring management schedules manually. In this paper, a novel proactive resource management approach is proposed which aims to predict VDI pool workload adaptively by utilizing CoArse to Fine historical dEscriptive (CAFE) features. Specifically, aggregate session count from pool end users serves as the basis for workload measurement and predictive model induction. Extensive experiments on real VDI customers data sets clearly validate the effectiveness of multi-grained features for VDI workload prediction. Furthermore, practical insights identified in our VDI data analytics are also discussed.