Hierarchical Feature Representation Research Articles

Outlier classification for microbiological open set recognition

Precise detection and effective intervention of pathogenic microorganisms are pivotal to ensure crop yield. Traditional open set recognition methods distinguish unknown classes, i.e., those not involved in training, as outliers or separate classes, which is insufficient for real-world applications like recognizing airborne microorganisms and particles. In this paper, a new setting for the classification of outliers for open set recognition is proposed. The proposed method classifies unknown classes while achieving the classification of known classes, treating each category in the open set equally. The two cores of the method are hierarchical feature representation and feature gallery construction. To enhance the feature representation, this paper proposes a novel architectural module of spatial attention on region hierarchy, namely SPARC. This module extracts features from hierarchical levels and models spatial interaction through the attention mechanism. With the induction of local-to-global description, minute differences with ambiguous appearances between classes can be detected, and the negative effects of image quality degradation can be ameliorated. Furthermore, a feature gallery containing the feature representation from both known and unknown classes is developed for feature retrieval. To demonstrate the effectiveness of the proposed model, the first dataset composed of agricultural microorganisms, named SMAD, is constructed, and experiments are conducted on existing datasets and the newly proposed SMAD dataset. The experimental results outperform the prior art on these datasets under both the outlier rejection and outlier classification settings.

Emotion-aware hierarchical interaction network for multimodal image aesthetics assessment

Image aesthetics assessment (IAA) has attracted increasing attention recently but is still challenging due to its high abstraction and complexity. Intuitively, image emotion and aesthetics are both human subjective feelings evoked by visual content. Previous researches have demonstrated that image aesthetics has intrinsic relationships with emotion. In fact, human emotional experience has potential impact on the perception of image aesthetics. Therefore, emotion information can be exploited to enhance aesthetic representation learning. Inspired by this, this paper presents an Emotion-Aware Hierarchical Interaction NETwork (EAHI-NET) for multimodal image aesthetics assessment, which explores both intra-modal and inter-modal interactions between aesthetics and emotions hierarchically. Specifically, we first propose the intra-modal emotion-aesthetics interaction module to obtain emotion-enhanced visual and textual aesthetic representations respectively. Then we propose the inter-modal feature enhancement to obtain the cross-modal aesthetic and emotion features. Finally, we design the inter-modal emotion-aesthetics interaction module to further investigate the cross-modal interplay between aesthetics and emotion, based on which hierarchical feature representations are achieved for multimodal IAA. The extensive experiments show that the proposed method can outperform the state-of-the-arts on multimodal AVA and Photo.net datasets.

Learning Aligned Vertex Convolutional Networks for Graph Classification.

Graph convolutional networks (GCNs) are powerful tools for graph structure data analysis. One main drawback arising in most existing GCN models is that of the oversmoothing problem, i.e., the vertex features abstracted from the existing graph convolution operation have previously tended to be indistinguishable if the GCN model has many convolutional layers (e.g., more than two layers). To address this problem, in this article, we propose a family of aligned vertex convolutional network (AVCN) models that focus on learning multiscale features from local-level vertices for graph classification. This is done by adopting a transitive vertex alignment algorithm to transform arbitrary-sized graphs into fixed-size grid structures. Furthermore, we define a new aligned vertex convolution operation that can effectively learn multiscale vertex characteristics by gradually aggregating local-level neighboring aligned vertices residing on the original grid structures into a new packed aligned vertex. With the new vertex convolution operation to hand, we propose two architectures for the AVCN models to extract different hierarchical multiscale vertex feature representations for graph classification. We show that the proposed models can avoid iteratively propagating redundant information between specific neighboring vertices, restricting the notorious oversmoothing problem arising in most spatial-based GCN models. Experimental evaluations on benchmark datasets demonstrate the effectiveness.

Learning high-order spatial interactions and cascaded attention decoding for MRI-based ventricular segmentation

The precise segmentation of ventricles facilitates the provision of quantitative information concerning cardiac structure and function to medical professionals, thereby enabling accurate diagnosis. The present challenges in ventricular segmentation are twofold: Capturing positional information and morphological features of the ventricular region is challenging due to the cyclic beating of the heart, and ventricles exhibit significant similarity to other tissues in greyscale values. Compared to Convolution neural networks (CNNs) that lack the ability to establish long-range dependencies, Transformer which could perceive global contextual information have made significant progress in ventricular segmentation. However, Vanilla Transformer neglects cross-channel dependencies that can be utilized to capture positional and morphological features of ventricles hidden in numerous channels. To address this problem, this work proposes Efficient Split Channel Product Net (ESCP-Net). Specifically, ESCP-Net designs a Split-Channel Product Encoding Module (SCPEM) to grasp the ventricular position and morphological information by means of High Order Spatial interactions. Moreover, ESCP-Net cascades the Non-isomorphic Feature Fusion Module (NFFM) and the attention mechanism, thereby fusing the hierarchical feature representations and enlarging the feature grey scale difference to generating clearer feature representation. To verify this proposed method, extensive experiments on three heart chambers segmentation datasets (Automated Cardiac Diagnosis Challenge[ACDC]2017, Right ventricle Segmentation Challenge[RVSC]2012, Challenge of Left Ventricular Segmentation in Heart MR[SunnyBrook]2009) are conducted. The proposed method achieved the best performance with a DSC of 91.66% in ACDC datasets and 85.54% in RVSC and 95.94% in SunnyBrook. This work demonstrated that our method provides a promising solution for ventricular Segmentation.

Pushing the Limit of Fine-Tuning for Few-Shot Learning: Where Feature Reusing Meets Cross-Scale Attention

Due to the scarcity of training samples, Few-Shot Learning (FSL) poses a significant challenge to capture discriminative object features effectively. The combination of transfer learning and meta-learning has recently been explored by pre-training the backbone features using labeled base data and subsequently fine-tuning the model with target data. However, existing meta-learning methods, which use embedding networks, suffer from scaling limitations when dealing with a few labeled samples, resulting in suboptimal results. Inspired by the latest advances in FSL, we further advance the approach of fine-tuning a pre-trained architecture by a strengthened hierarchical feature representation. The technical contributions of this work include: 1) a hybrid design named Intra-Block Fusion (IBF) to strengthen the extracted features within each convolution block; and 2) a novel Cross-Scale Attention (CSA) module to mitigate the scaling inconsistencies arising from the limited training samples, especially for cross-domain tasks. We conducted comprehensive evaluations on standard benchmarks, including three in-domain tasks (miniImageNet, CIFAR-FS, and FC100), as well as two cross-domain tasks (CDFSL and Meta-Dataset). The results have improved significantly over existing state-of-the-art approaches on all benchmark datasets. In particular, the FSL performance on the in-domain FC100 dataset is more than three points better than the latest PMF (Hu et al. 2022).

Enhancing Disease Prediction with a Hybrid CNN-LSTM Framework in EHRs

This study developed a novel hybrid deep learning framework aimed at enhancing the accuracy of disease prediction using temporal data from Electronic Health Records (EHRs). The framework integrates Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) networks, leveraging the strength of CNNs in extracting hierarchical feature representations from complex data and the capability of LSTMs in capturing long-term dependencies in temporal information. An empirical investigation on real-world EHR datasets revealed that, compared to Support Vector Machine (SVM) models, standalone CNNs, and LSTMs, this hybrid deep learning network demonstrated significantly higher prediction accuracy in disease prediction tasks. This research not only advances the performance of predictive models in the health data analytics domain but also underscores the importance of adopting and further developing advanced deep learning technologies to address the complexity of modern medical data. Our findings advocate for a shift towards integrating complex neural network architectures in developing predictive models, potentially offering avenues for more personalized and proactive disease management and care, thereby setting new standards for future health management practices.

Comparison and analysis of the number of convolutional layers in CNN-based breast cancer detection

Breast cancer is becoming more and more common, and the mortality rate is increasing, which is indeed a serious problem. To alleviate this problem, this study applies a convolutional neural network (CNN) model to analyze breast cancer images formed by mammography. The CNN model has the property of automatically learning hierarchical feature representations from the original image data. The structure of the CNN enables it to capture spatially hierarchical features at multiple scales, including edges, textures, and objects. Specifically, this study explores the performance of CNN models on prediction tasks by varying the number of convolutional layers. This study is conducted on the DDSM mammography dataset. The experimental results demonstrate that CNN models are effective for detecting breast cancer, both in terms of accuracy and loss rate, and that more convolutional layers improve performance. In particular, the model saturates at four convolutional layers to reach the highest performance. Thus, this study helps to accelerate the efficiency of breast cancer detection and paves the way for more efficient methods in the future.

QGFormer: Queries-guided transformer for flexible medical image synthesis with domain missing

Domain missing poses a common challenge in medical clinical practice, limiting diagnostic accuracy compared to the complete multi-domain images that provide complementary information. We propose QGFormer to address this issue by flexibly imputing missing domains from any available source domain using a single model, which is challenging due to (1) the inherent limitation of CNNs to capture long-range dependencies, (2) the difficulty in modeling the inter- and intra-domain dependencies of multi-domain images, and (3) inefficiencies in fusing domain-specific features associated with missing domains. To tackle these challenges, we introduce two spatial-domanial attentions (SDAs), which establish intra-domain (spatial dimension) and inter-domain (domain dimension) dependencies independently or jointly. QGFormer, constructed based on SDAs, comprises three components: Encoder, Decoder and Fusion. The Encoder and Decoder form the backbone, modeling contextual dependencies to create a hierarchical representation of features. The QGFormer Fusion then adaptively aggregates these representations to synthesize specific missing domains from coarse to fine, guided by learnable domain queries. This process is interpretable because the attention scores in Fusion indicate how much attention the target domains pay to different inputs and regions. In addition, the scalable architecture enables QGFormer to segment tumors with domain missing by replacing domain queries with segment queries. Extensive experiments demonstrate that our approach achieves consistent improvements in multi-domain imputation, cross-domain image translation and multitask of synthesis and segmentation.

Cross-language semantic-affective interaction – with evidence from Chinese EFL learners

Abstract Semantic and affective priming have long been treated separately in psycholinguistic studies. Recently, however, the question of whether and how these two primings interact has become controversial, especially in cross-language contexts where such discussions are rare. In the present study, four mixed-design experiments were conducted with Chinese EFL learners to investigate cross-language semantic-affective interactions: 3 (prime valence: negative, positive, neutral) × 2 (semantic relatedness: related, unrelated). Results show that semantic priming effects occurred in the L1L1 and L1L2 conditions, whereas affective priming effects were observed in the L2L2 condition. In the L2L1 priming condition, only emotion primes induced cross-language priming. These results suggest that semantic and emotional accesses are activated automatically and separately, but can facilitate cross-language word processing mutually. The results support the hierarchical representation of semantic features of emotion words from L1 to L2 in the unbalanced bilingual mental lexicon, while affective attributes are spread across a distributed network.

Unveiling the Complexity of Medical Imaging through Deep Learning Approaches

Recent advancements in deep learning, particularly convolutional networks, have rapidly become the preferred methodology for analyzing medical images, facilitating tasks like disease segmentation, classification, and pattern quantification. Central to these advancements is the capacity to leverage hierarchical feature representations acquired solely from data. This comprehensive review meticulously examines a variety of deep learning techniques applied across diverse healthcare domains, delving into the intricate realm of medical imaging to unveil concealed patterns through strategic deep learning methodologies. Encompassing a range of diseases, including Alzheimer’s, breast cancer, brain tumors, glaucoma, heart murmurs, retinal microaneurysms, colorectal liver metastases, and more, the analysis emphasizes contributions succinctly summarized in a tabular form.The table provides an overview of various deep learning approaches applied to different diseases, incorporating methodologies, datasets, and outcomes for each condition.Notably, performance metrics such as accuracy, specificity, sensitivity, and other crucial measures underscore the achieved results. Specifically, an in-depth discussion is conducted on the Convolutional Neural Network (CNN) owing to its widespread adoption as a paramount tool in computer vision tasks. Moreover, an exhaustive exploration encompasses deep learning classification approaches, procedural aspects of medical image processing, as well as a thorough examination of key features and characteristics. At the end, we delve into a range of research challenges and put forth potential avenues for future improvements in the field.

MRAN: Multimodal relationship-aware attention network for fake news detection

Existing multimodal fake news detection methods face challenges in jointly capturing the intramodality and cross-modal correlation relationships between image regions and text fragments. Additionally, these methods lack comprehensive hierarchical semantics mining for text. These limitations result in ineffective utilization of multimodal information and impact detection performance. To address these issues, we propose a multimodal relationship-aware attention network (MRAN), which consists of three main steps. First, a multi-level encoding network is employed to extract hierarchical semantic feature representations of text, while the visual feature extractor VGG19 learns image feature representations. Second, the captured text and image representations are input into the relationship-aware attention network, which generates high-order fusion features by calculating the similarity between information segments within modalities and cross-modal similarity. Finally, the fusion features are passed through a fake news detector, which identifies fake news. Experimental results on three benchmark datasets demonstrate the effectiveness of MRAN, highlighting its strong detection performance.

Unleashing the full potential of hyperspectral imaging: Decoupled image and frequency-domain spatial–spectral framework

Hyperspectral image classification (HSIC) is a rapidly developing field that utilizes deep learning methods. However, the reliance on convolutional neural networks (CNNs) for spectral–spatial feature extraction presents certain limitations. Specifically, the use of the fixed-position convolutional kernels in CNNs hinders their ability to capture fully the spectral information around the spatial central pixel, thereby overlooking critical differences between features. To address this issue, a decoupled image- and frequency-domain spectral–spatial framework for HSIC was developed in this study. This method incorporates image- and frequency-domain-based multiscale learnable convolutional attention to refine the differentiating features of the different feature distributions. Additionally, a novel frequency-domain information enhancement module was designed to extract the structural shape and texture details under the semantic constraints of the frequency phase, complementing the image domain to improve the extracted feature maps. Furthermore, a simple and efficient hierarchical feature representation module was introduced to extract both local and global information effectively from the fused features. The experimental results obtained using three open datasets and a practical hyperspectral image of the Gaofen-5 satellite demonstrate that the proposed method outperforms other state-of-the-art HSIC methods.

Research on the detection model of mental illness of online forum users based on convolutional network

Recently, there will be more than 4.62 billion social media users worldwide. A large number of users tend to publish personal emotional dynamics or express opinions on social media. These massive user data provide data support for the development of mental illness detection research and have achieved good results. However, it is difficult for current mental illness detection models to accurately identify key emotional features from a large number of posts issued by users to detect problem users. In view of the fact that the existing models cannot more accurately extract the words with high emotional contribution in the content of user posts, this paper proposes two hierarchical user post feature representation models, named Single-Gated LeakReLU-CNN (SGL-CNN) and Multi-Gated LeakyReLU-CNN (MGL-CNN). We leverage these 2 models to identify users with mental illness in online forums. For all posts published by each user within a certain time span, the model proposed in this paper can identify key emotional features in them and filter out other unimportant information as much as possible. In addition, the addition of gating units in this paper can significantly improve the performance of emotion detection tasks. The experimental results based on the task of RSDD dataset prove that the performance of the model proposed in this paper is superior to that of the existing methods.

CartiMorph: A framework for automated knee articular cartilage morphometrics

We introduce CartiMorph, a framework for automated knee articular cartilage morphometrics. It takes an image as input and generates quantitative metrics for cartilage subregions, including the percentage of full-thickness cartilage loss (FCL), mean thickness, surface area, and volume. CartiMorph leverages the power of deep learning models for hierarchical image feature representation. Deep learning models were trained and validated for tissue segmentation, template construction, and template-to-image registration. We established methods for surface-normal-based cartilage thickness mapping, FCL estimation, and rule-based cartilage parcellation. Our cartilage thickness map showed less error in thin and peripheral regions. We evaluated the effectiveness of the adopted segmentation model by comparing the quantitative metrics obtained from model segmentation and those from manual segmentation. The root-mean-squared deviation of the FCL measurements was less than 8%, and strong correlations were observed for the mean thickness (Pearson’s correlation coefficient ρ∈[0.82,0.97]), surface area (ρ∈[0.82,0.98]) and volume (ρ∈[0.89,0.98]) measurements. We compared our FCL measurements with those from a previous study and found that our measurements deviated less from the ground truths. We observed superior performance of the proposed rule-based cartilage parcellation method compared with the atlas-based approach. CartiMorph has the potential to promote imaging biomarkers discovery for knee osteoarthritis.

Region-Aware Hierarchical Latent Feature Representation Learning-Guided Clustering for Hyperspectral Band Selection.

Hyperspectral band selection aims to identify an optimal subset of bands for hyperspectral images (HSIs). For most existing clustering-based band selection methods, they directly stretch each band into a single feature vector and employ the pixelwise features to address band redundancy. In this way, they do not take full consideration of the spatial information and deal with the importance of different regions in HSIs, which leads to a nonoptimal selection. To address these issues, a region-aware hierarchical latent feature representation learning-guided clustering (HLFC) method is proposed. Specifically, in order to fully preserve the spatial information of HSIs, the superpixel segmentation algorithm is adopted to segment HSIs into multiple regions first. For each segmented region, the similarity graph is constructed to reflect the bands-wise similarity, and its corresponding Laplacian matrix is generated for learning low-dimensional latent features in a hierarchical way. All latent features are then fused to form a unified feature representation of HSIs. Finally, k -means clustering is utilized on the unified feature representation matrix to generate multiple clusters from which the band with maximum information entropy is selected to form the final subset of bands. Extensive experimental results demonstrate that the proposed clustering method can achieve superior performance than the state-of-the-art representative methods on the band selection. The demo code of this work is publicly available at https://github.com/WangJun2023/HLFC.

Speech Emotion Recognition Using Convolution Neural Networks and Multi-Head Convolutional Transformer.

Speech emotion recognition (SER) is a challenging task in human-computer interaction (HCI) systems. One of the key challenges in speech emotion recognition is to extract the emotional features effectively from a speech utterance. Despite the promising results of recent studies, they generally do not leverage advanced fusion algorithms for the generation of effective representations of emotional features in speech utterances. To address this problem, we describe the fusion of spatial and temporal feature representations of speech emotion by parallelizing convolutional neural networks (CNNs) and a Transformer encoder for SER. We stack two parallel CNNs for spatial feature representation in parallel to a Transformer encoder for temporal feature representation, thereby simultaneously expanding the filter depth and reducing the feature map with an expressive hierarchical feature representation at a lower computational cost. We use the RAVDESS dataset to recognize eight different speech emotions. We augment and intensify the variations in the dataset to minimize model overfitting. Additive White Gaussian Noise (AWGN) is used to augment the RAVDESS dataset. With the spatial and sequential feature representations of CNNs and the Transformer, the SER model achieves 82.31% accuracy for eight emotions on a hold-out dataset. In addition, the SER system is evaluated with the IEMOCAP dataset and achieves 79.42% recognition accuracy for five emotions. Experimental results on the RAVDESS and IEMOCAP datasets show the success of the presented SER system and demonstrate an absolute performance improvement over the state-of-the-art (SOTA) models.

DeepCNN: Spectro‐temporal feature representation for speech emotion recognition

AbstractSpeech emotion recognition (SER) is an important research problem in human‐computer interaction systems. The representation and extraction of features are significant challenges in SER systems. Despite the promising results of recent studies, they generally do not leverage progressive fusion techniques for effective feature representation and increasing receptive fields. To mitigate this problem, this article proposes DeepCNN, which is a fusion of spectral and temporal features of emotional speech by parallelising convolutional neural networks (CNNs) and a convolution layer‐based transformer. Two parallel CNNs are applied to extract the spectral features (2D‐CNN) and temporal features (1D‐CNN) representations. A 2D‐convolution layer‐based transformer module extracts spectro‐temporal features and concatenates them with features from parallel CNNs. The learnt low‐level concatenated features are then applied to a deep framework of convolutional blocks, which retrieves high‐level feature representation and subsequently categorises the emotional states using an attention gated recurrent unit and classification layer. This fusion technique results in a deeper hierarchical feature representation at a lower computational cost while simultaneously expanding the filter depth and reducing the feature map. The Berlin Database of Emotional Speech (EMO‐BD) and Interactive Emotional Dyadic Motion Capture (IEMOCAP) datasets are used in experiments to recognise distinct speech emotions. With efficient spectral and temporal feature representation, the proposed SER model achieves 94.2% accuracy for different emotions on the EMO‐BD and 81.1% accuracy on the IEMOCAP dataset respectively. The proposed SER system, DeepCNN, outperforms the baseline SER systems in terms of emotion recognition accuracy on the EMO‐BD and IEMOCAP datasets.

Dynamic Cross-Task Representation Adaptation for Clinical Targets Co-Segmentation in CT Image-Guided Post-Prostatectomy Radiotherapy.

Adjuvant and salvage radiotherapy after radical prostatectomy requires precise delineations of prostate bed (PB), i.e., the clinical target volume, and surrounding organs at risk (OARs) to optimize radiotherapy planning. Segmenting PB is particularly challenging even for clinicians, e.g., from the planning computed tomography (CT) images, as it is an invisible/virtual target after the operative removal of the cancerous prostate gland. Very recently, a few deep learning-based methods have been proposed to automatically contour non-contrast PB by leveraging its spatial reliance on adjacent OARs (i.e., the bladder and rectum) with much more clear boundaries, mimicking the clinical workflow of experienced clinicians. Although achieving state-of-the-art results from both the clinical and technical aspects, these existing methods improperly ignore the gap between the hierarchical feature representations needed for segmenting those fundamentally different clinical targets (i.e., PB and OARs), which in turn limits their delineation accuracy. This paper proposes an asymmetric multi-task network integrating dynamic cross-task representation adaptation (i.e., DyAdapt) for accurate and efficient co-segmentation of PB and OARs in one-pass from CT images. In the learning-to-learn framework, the DyAdapt modules adaptively transfer the hierarchical feature representations from the source task of OARs segmentation to match up with the target (and more challenging) task of PB segmentation, conditioned on the dynamic inter-task associations learned from the learning states of the feed-forward path. On a real-patient dataset, our method led to state-of-the-art results of PB and OARs co-segmentation. Code is available at https://github.com/ladderlab-xjtu/DyAdapt.

F2Trans: High-Frequency Fine-Grained Transformer for Face Forgery Detection

In recent years, face forgery detectors have aroused great interest and achieved impressive performance, but they are still struggling with generalization and robustness. In this work, we explore taking full advantage of the fine-grained forgery traces in both spatial and frequency domains to alleviate this issue. Specifically, we propose a novel High-Frequency Fine-Grained Transformer (F2Trans) network which contains two important components, namely Central Difference Attention (CDA) and High-frequency Wavelet Sampler (HWS). The premier CDA module is capable of capturing invariant fine-grained manipulation patterns by aggregating both pixel-level intensity and gradient information of the query to generate key and value pairs. Subsequently, the proposed HWS discards the low-frequency components of wavelet transformation and hierarchically explores high-frequency forgery cues of feature maps, which prevents model confusion caused by low-frequency components and pays attention to local frequency information. In addition, HWS can be employed as a special pooling layer for the F2Trans architecture to produce hierarchical feature representations in the spatial-frequency domain. Extensive experiments on multiple popular benchmarks demonstrate the generalization and robustness of the specially designed F2Trans framework is well-tailored for face forgery detection when confronting the cross-dataset, cross-manipulation, and unseen perturbations.

Hyperspectral image classification based on three-dimensional adaptive sampling and improved iterative shrinkage-threshold algorithm

Abundant spectral information of hyperspectral images (HSI) provides rich information for HSI classification, which often brings high dimensional data resulting in the dilemma between the demand for fine data and the limited resources such as computation, storage as well as transmission band-width. To address this issue, we propose a deep hierarchical feature representation model based on three-dimensional adaptive sampling and improved iterative shrinkage-threshold algorithm (ISTA) for HSI classification. Due to the adaptive sampling, we improve ISTA with deep learning network for spectral–spatial feature representation since the ISTA is no longer applicable for the sampled data reconstruction. Through end-to-end joint learning, the proposed method can not only effectively reduce the required data, but also learn discriminative features for HSI classification, which will be meaningful for the HSI’s transmission from the space satellites and fast classification. Experimental results demonstrate the effectiveness and superiority of the proposed method on three public HSI datasets.