Distribution In Feature Space Research Articles

Bidirectional consistency with temporal-aware for semi-supervised time series classification

Semi-supervised learning (SSL) has achieved significant success due to its capacity to alleviate annotation dependencies. Most existing SSL methods utilize pseudo-labeling to propagate useful supervised information for training unlabeled data. However, these methods ignore learning temporal representations, making it challenging to obtain a well-separable feature space for modeling explicit class boundaries. In this work, we propose a semi-supervised Time Series classification framework via Bidirectional Consistency with Temporal-aware (TS-BCT), which regularizes the feature space distribution by learning temporal representations through pseudo-label-guided contrastive learning. Specifically, TS-BCT utilizes time-specific augmentation to transform the entire raw time series into two distinct views, avoiding sampling bias. The pseudo-labels for each view, generated through confidence estimation in the feature space, are then employed to propagate class-related information into unlabeled samples. Subsequently, we introduce a temporal-aware contrastive learning module that learns discriminative temporal-invariant representations. Finally, we design a bidirectional consistency strategy by incorporating pseudo-labels from two distinct views into temporal-aware contrastive learning to construct a class-related contrastive pattern. This strategy enables the model to learn well-separated feature spaces, making class boundaries more discriminative. Extensive experimental results on real-world datasets demonstrate the effectiveness of TS-BCT compared to baselines.

Embedded prompt tuning: Towards enhanced calibration of pretrained models for medical images

Foundation models pre-trained on large-scale data have been widely witnessed to achieve success in various natural imaging downstream tasks. Parameter-efficient fine-tuning (PEFT) methods aim to adapt foundation models to new domains by updating only a small portion of parameters in order to reduce computational overhead. However, the effectiveness of these PEFT methods, especially in cross-domain few-shot scenarios, e.g., medical image analysis, has not been fully explored. In this work, we facilitate the study of the performance of PEFT when adapting foundation models to medical image classification tasks. Furthermore, to alleviate the limitations of prompt introducing ways and approximation capabilities on Transformer architectures of mainstream prompt tuning methods, we propose the Embedded Prompt Tuning (EPT) method by embedding prompt tokens into the expanded channels. We also find that there are anomalies in the feature space distribution of foundation models during pre-training process, and prompt tuning can help mitigate this negative impact. To explain this phenomenon, we also introduce a novel perspective to understand prompt tuning: Prompt tuning is a distribution calibrator. And we support it by analysing patch-wise scaling and feature separation operations contained in EPT. Our experiments show that EPT outperforms several state-of-the-art fine-tuning methods by a significant margin on few-shot medical image classification tasks, and completes the fine-tuning process within highly competitive time, indicating EPT is an effective PEFT method. The source code is available at github.com/zuwenqiang/EPT.

A deep attention model for wide-genome protein-peptide binding affinity prediction at a sequence level

Peptides are pivotal in numerous biological activities by engaging in up to 40 % of protein-protein interactions in many cellular processes. Due to their exceptional specificity and effectiveness, peptides have emerged as promising candidates for drug design. However, accurately predicting protein-peptide binding affinity remains a challenging. Aiming at the problem, we develop a prediction model PepPAP based on convolutional neural network and multi-head attention, which relies solely on sequence features. These features include physicochemical properties, intrinsic disorder, sequence encoding, and especially interface propensity which is extracted from 16,689 non-redundant protein-peptide complexes. Notably, the adopted regression stratification cross-validation scheme proposed in our previous work is beneficial to improve the prediction for the cases with extreme binding affinity values. On three benchmark test datasets: T100, a series of peptides targeting to PDZ domain and CXCR4, PepPAP shows excellent performance, outperforming the existing methods and demonstrating its good generalization ability. Furthermore, PepPAP has good results in binary interaction prediction, and the analysis of the feature space distribution visualization highlights PepPAP's effectiveness. To the best of our knowledge, PepPAP is the first sequence-based deep attention model for wide-genome protein-peptide binding affinity prediction, and holds the potential to offer valuable insights for the peptide-based drug design.

Discriminative Action Snippet Propagation Network for Weakly Supervised Temporal Action Localization

Weakly supervised temporal action localization (WTAL) aims to classify and localize actions in untrimmed videos with only video-level labels. Recent studies have attempted to obtain more accurate temporal boundaries by exploiting latent action instances in ambiguous snippets or propagating representative action features. However, empirically handcrafted ambiguous snippet extraction and the imprecise alignment of representative snippet propagation lead to challenges in modeling the completeness of actions for these methods. In this article, we propose a Discriminative Action Snippet Propagation Network (DASP-Net) to accurately discover ambiguous snippets in videos and propagate discriminative instance-level features throughout the video for improving action completeness. Specifically, we introduce a novel discriminative feature propagation module for capturing the global contextual attention and propagating the action concept across the whole video by perceiving the discriminative action snippets with instance information from the same video. Simultaneously, we incorporate denoised pseudo-labels as supervision, where we correct the controversial prediction based on the feature space distribution during training, thereby alleviating false detection caused by noise background features. Furthermore, we design an ambiguous feature mining module, which maximizes the feature affinity information of action and background in ambiguous snippets to generate more accurate latent action and background snippets and learns more precise action instance boundaries through contrastive learning of action and background snippets. Extensive experiments show that DASP-Net achieves state-of-the-art results on THUMOS14 and ActivityNet1.2 datasets.

A hybrid data driven framework considering feature extraction for battery state of health estimation and remaining useful life prediction

Battery life prediction is of great significance to the safe operation, and reduces the maintenance costs. This paper proposes a hybrid framework considering feature extraction to achieve more accurate and stable life prediction performance of the battery. By feature extraction, eight features are obtained to fed into the life prediction model. The hybrid framework combines variational mode decomposition, the multi-kernel support vector regression model and the improved sparrow search algorithm to solve the problem of data backward, uneven distribution of high-dimensional feature space and the local escape ability, respectively. Better parameters of the estimation model are obtained by introducing the elite chaotic opposition-learning strategy and adaptive weights to optimize the sparrow search algorithm. The algorithm can improve the local escape ability and convergence performance and find the global optimum. The comparison is conducted by dataset from National Aeronautics and Space Administration which shows that the proposed framework has a more accurate and stable prediction performance. Compared with other algorithms, the SOH estimation accuracy of the proposed algorithm is improved by 0.16%–1.67%. With the advance of the start point, the RUL prediction accuracy of the proposed algorithm does not change much.

Ship Infrared Automatic Target Recognition Based on Bipartite Graph Recommendation: A Model-Matching Method

Deep learning technology has greatly propelled the development of intelligent and information-driven research on ship infrared automatic target recognition (SIATR). In future scenarios, there will be various recognition models with different mechanisms to choose from. However, in complex and dynamic environments, ship infrared (IR) data exhibit rich feature space distribution, resulting in performance variations among SIATR models, thus preventing the existence of a universally superior model for all recognition scenarios. In light of this, this study proposes a model-matching method for SIATR tasks based on bipartite graph theory. This method establishes evaluation criteria based on recognition accuracy and feature learning credibility, uncovering the underlying connections between IR attributes of ships and candidate models. The objective is to selectively recommend the optimal candidate model for a given sample, enhancing the overall recognition performance and applicability of the model. We separately conducted tests for the optimization of accuracy and credibility on high-fidelity simulation data, achieving Accuracy and EDMS (our credibility metric) of 95.86% and 0.7781. Our method improves by 1.06% and 0.0274 for each metric compared to the best candidate models (six in total). Subsequently, we created a recommendation system that balances two tasks, resulting in improvements of 0.43% (accuracy) and 0.0071 (EDMS). Additionally, considering the relationship between model resources and performance, we achieved a 28.35% reduction in memory usage while realizing enhancements of 0.33% (accuracy) and 0.0045 (EDMS).

Dual Balanced Class-Incremental Learning With im-Softmax and Angular Rectification.

Owing to the superior performances, exemplar-based methods with knowledge distillation (KD) are widely applied in class incremental learning (CIL). However, it suffers from two drawbacks: 1) data imbalance between the old/learned and new classes causes the bias of the new classifier toward the head/new classes and 2) deep neural networks (DNNs) suffer from distribution drift when learning sequence tasks, which results in narrowed feature space and deficient representation of old tasks. For the first problem, we analyze the insufficiency of softmax loss when dealing with the problem of data imbalance in theory and then propose the imbalance softmax (im-softmax) loss to relieve the imbalanced data learning, where we re-scale the output logits to underfit the head/new classes. For another problem, we calibrate the feature space by incremental-adaptive angular margin (IAAM) loss. The new classes form a complete distribution in feature space yet the old are squeezed. To recover the old feature space, we first compute the included angle of normalized features and normalized anchor prototypes, and use the angle distribution to represent the class distribution, then we replenish the old distribution with the deviation from the new. Each anchor prototype is predefined as a learnable vector for a designated class. The proposed im-softmax reduces the bias in the linear classification layer. IAAM rectifies the representation learning, reduces the intra-class distance, and enlarges the inter-class margin. Finally, we seamlessly combine the im-softmax and IAAM in an end-to-end training framework, called the dual balanced class incremental learning (DBL), for further improvements. Experiments demonstrate the proposed method achieves state-of-the-art (SOTA) performances on several benchmarks, such as CIFAR10, CIFAR100, Tiny-ImageNet, and ImageNet-100.

A domain adversarial graph convolutional network for intelligent monitoring of tool wear in machine tools

The prediction of tool wear on CNC machine tools holds great significance for enhancing both the safety of tool machining and the quality of product machining. The current data-driven prediction methods for tool wear in machining have yielded very promising results. However, tool wear prediction models for the same machine tool under different machining conditions lack universality since the data feature space distribution under varying work conditions is different. To bridge the gap between source and target domains, data structure information is all too commonly ignored beyond domain and class labels. To attack these problems, a transfer learning method for acquiring domain-invariant features of tool wear based on a dynamic domain adversarial graph convolutional network (DDAGCN) is proposed in this paper. The network integrates three effective alignment mechanisms - domain, data structure, and class centroid alignments - and evaluates the relative importance of the marginal and conditional distributions quantitatively. In addition, a dynamic factor is introduced to quantitatively assess the relative significance of marginal and conditional distributions, which effectively facilitate the learning of domain-invariant features and decrease differences between source and target domains. The superiority of the proposed method was validated through experiments under two distinct conditions.

Does protein pretrained language model facilitate the prediction of protein-ligand interaction?

Protein-ligand interaction (PLI) is a critical step for drug discovery. Recently, protein pretrained language models (PLMs) have showcased exceptional performance across a wide range of protein-related tasks. However, a significant heterogeneity exists between the PLM and PLI tasks, leading to a degree of uncertainty. In this study, we propose a method that quantitatively assesses the significance of protein PLMs in PLI prediction. Specifically, we analyze the performance of three widely-used protein PLMs (TAPE, ESM-1b, and ProtTrans) on three PLI tasks (PDBbind, Kinase, and DUD-E). The model with pre-training consistently achieves improved performance and decreased time cost, demonstrating that enhance both the accuracy and efficiency of PLI prediction. By quantitatively assessing the transferability, the optimal PLM for each PLI task is identified without the need for costly transfer experiments. Additionally, we examine the contributions of PLMs on the distribution of feature space, highlighting the improved discriminability after pre-training. Our findings provide insights into the mechanisms underlying PLMs in PLI prediction and pave the way for the design of more interpretable and accurate PLMs in the future. Code and data are freely available at https://github.com/brian-zZZ/PLM-PLI.

PulseID: Multi-scale photoplethysmographic identification using a deep convolutional neural network

Photoplethysmography (PPG) signals reflect the cardiovascular health of the human body and contain discriminate information that can be utilized as biometric traits in the personal human identification field. Recent approaches achieve it by hand-crafted features and using shallow networks to extract features, which might not achieve comparable results on large-scale datasets. In this paper, we first propose PPG augmentation to generate multi-scale PPG signals to avoid overfitting. Subsequently, a deep end-to-end model, namely PulseID, is proposed to extract multi-scale and unique features from PPG signals. Finally, a curriculum learning guided training strategy is proposed to provide elaborate supervisions for PulseID. As key modules in curriculum learning, contrastive learning is adopted to enhance the discriminate distribution in feature space. Comprehensive experiments are demonstrated on 4 databases which are ZJPhys, Capno, Bidmc, and Biosec2 respectively to show our superior performance on both intra- and cross-dataset testings. Specifically, on intra-dataset testing, the proposed PulseID achieves superior performance on the largest database compared with state-of-the-art methods which are 94.3% and 2.9% in terms of Average Accuracy and Equal Error Rate. On cross-dataset testing, PulseID also achieves comparable performance on unknown individual recognition regardless of various sampling rates, races, and body conditions. In conclusion, without heavy pre-processing, the proposed PulseID can be easily used in known- and unknown-individual recognition with superior performance.

Learning correlation information for multi-label feature selection

In many real-world multi-label applications, the content of multi-label data is usually characterized by high dimensional features, which contains complex correlation information, i.e., label correlations and redundant features. To alleviate the problem, we present a novel scheme, called learning correlation information for multi-label feature selection (LCIFS) method, by jointly digging up label correlations and controlling feature redundancy. To be specific, the regression model via manifold framework is presented to fit the relationship between feature space and label distribution, during which adaptive spectral graph is leveraged to learn more precise structural correlations of labels simultaneously. Besides, we utilize the relevance of features to constrain the redundancy of the generated feature subset, and a general ℓ2,p-norm regularized model is employed to fulfill more robust feature selection. The proposed method is transformed into an explicit optimization function, which is conquered by an efficient iterative optimization algorithm. Finally, we conduct comprehensive experiments on twelve realistic multi-label datasets, including text domain, image domain, and audio domain. The statistic results demonstrate the effectiveness and superiority of the proposed method among nine competition methods.

Deep transfer learning for automatic speech recognition: Towards better generalization

Automatic speech recognition (ASR) has recently become an important challenge when using deep learning (DL). It requires large-scale training datasets and high computational and storage resources. Moreover, DL techniques and machine learning (ML) approaches in general, hypothesize that training and testing data come from the same domain, with the same input feature space and data distribution characteristics. This assumption, however, is not applicable in some real-world artificial intelligence (AI) applications. Moreover, there are situations where gathering real data is challenging, expensive, or rarely occurring, which cannot meet the data requirements of DL models. deep transfer learning (DTL) has been introduced to overcome these issues, which helps develop high-performing models using real datasets that are small or slightly different but related to the training data. This paper presents a comprehensive survey of DTL-based ASR frameworks to shed light on the latest developments and helps academics and professionals understand current challenges. Specifically, after presenting the DTL background, a well-designed taxonomy is adopted to inform the state-of-the-art. A critical analysis is then conducted to identify the limitations and advantages of each framework. Moving on, a comparative study is introduced to highlight the current challenges before deriving opportunities for future research.

Generalized Transfer Extreme Learning Machine for Unsupervised Cross-Domain Fault Diagnosis With Small and Imbalanced Samples

In most engineering scenarios, the collected datasets typically have small and imbalanced characteristics, and usually possess different data distributions between variable working conditions. Thus, several existing diagnosis models trained by small and imbalanced samples under one working condition may suffer from low accuracy for fault data under another working condition. To effectively alleviate such problem, a novel domain adaptation method, referred to as generalized transfer extreme learning machine (G-TELM), is proposed in this study. First, in a random feature space, marginal and conditional distribution discrepancy between source and target domains is minimized based on projected maximum mean discrepancy. Second, in order to give varied attention to different faulty samples, a cost-sensitive source domain classifier is utilized to reduce the difference between types of misclassification loss for small and imbalanced source data. Third, misclassification loss of target data is used to minimize the difference between the network’s outputs and prediction labels of target data. Last, L2-norm regularization is leveraged to constrain network output weights, and they are iteratively updated to optimize the transferability of model G-TELM. Experimental results of 168 transfer tasks with varying difficulty verify that the proposed model G-TELM can achieve an accurate diagnosis and outperform several state-of-art domain adaptation methods.

PKA2-Net: Prior Knowledge-Based Active Attention Network for Accurate Pneumonia Diagnosis on Chest X-Ray Images.

To accurately diagnose pneumonia patients on a limited annotated chest X-ray image dataset, a prior knowledge-based active attention network (PKA2-Net1) was constructed. The PKA2-Net uses improved ResNet as the backbone network and consists of residual blocks, novel subject enhancement and background suppression (SEBS) blocks and candidate template generators, where template generators are designed to generate candidate templates for characterizing the importance of different spatial locations in feature maps. The core of PKA2-Net is SEBS block, which is proposed based on the prior knowledge that highlighting distinctive features and suppressing irrelevant features can improve the recognition effect. The purpose of SEBS block is to generate active attention features without any high-level features and enhance the ability of the model to localize lung lesions. In SEBS block, first, a series of candidate templates T with different spatial energy distributions are generated and the controllability of the energy distribution in T enables active attention features to maintain the continuity and integrity of the feature space distributions. Second, Top-n templates are selected from T according to certain learning rules, which are then operated by a convolution layer for generating supervision information that can guide the inputs of SEBS block to form active attention features. We evaluated the PKA2-Net on the binary classification problem of identifying pneumonia and healthy controls on a dataset containing 5856 chest X-ray images (ChestXRay2017), the results showed that our method can achieve 97.63% accuracy and 0.9872 sensitivity.

A Multimodal Protein Representation Framework for Quantifying Transferability Across Biochemical Downstream Tasks.

Proteins are the building blocks of life, carrying out fundamental functions in biology. In computational biology, an effective protein representation facilitates many important biological quantifications. Most existing protein representation methods are derived from self-supervised language models designed for text analysis. Proteins, however, are more than linear sequences of amino acids. Here, a multimodal deep learning framework for incorporating ≈1 million protein sequence, structure, and functional annotation (MASSA) is proposed. A multitask learning process with five specific pretraining objectives is presented to extract a fine-grained protein-domain feature. Through pretraining, multimodal protein representation achieves state-of-the-art performance in specific downstream tasks such as protein properties (stability and fluorescence), protein-protein interactions (shs27k/shs148k/string/skempi), and protein-ligand interactions (kinase, DUD-E), while achieving competitive results in secondary structure and remote homology tasks. Moreover, a novel optimal-transport-based metric with rich geometry awareness is introduced to quantify the dynamic transferability from the pretrained representation to the related downstream tasks, which provides a panoramic view of the step-by-step learning process. The pairwise distances between these downstream tasks are also calculated, and a strong correlation between the inter-task feature space distributions and adaptability is observed.

Novel Task-Based Unification and Adaptation (TUA) Transfer Learning Approach for Bilingual Emotional Speech Data

Modern developments in machine learning methodology have produced effective approaches to speech emotion recognition. The field of data mining is widely employed in numerous situations where it is possible to predict future outcomes by using the input sequence from previous training data. Since the input feature space and data distribution are the same for both training and testing data in conventional machine learning approaches, they are drawn from the same pool. However, because so many applications require a difference in the distribution of training and testing data, the gathering of training data is becoming more and more expensive. High performance learners that have been trained using similar, already-existing data are needed in these situations. To increase a model’s capacity for learning, transfer learning involves transferring knowledge from one domain to another related domain. To address this scenario, we have extracted ten multi-dimensional features from speech signals using OpenSmile and a transfer learning method to classify the features of various datasets. In this paper, we emphasize the importance of a novel transfer learning system called Task-based Unification and Adaptation (TUA), which bridges the disparity between extensive upstream training and downstream customization. We take advantage of the two components of the TUA, task-challenging unification and task-specific adaptation. Our algorithm is studied using the following speech datasets: the Arabic Emirati-accented speech dataset (ESD), the English Speech Under Simulated and Actual Stress (SUSAS) dataset and the Ryerson Audio-Visual Database of Emotional Speech and Song dataset (RAVDESS). Using the multidimensional features and transfer learning method on the given datasets, we were able to achieve an average speech emotion recognition rate of 91.2% on the ESD, 84.7% on the RAVDESS and 88.5% on the SUSAS datasets, respectively.

Cluster Validity Index for Uncertain Data Based on a Probabilistic Distance Measure in Feature Space

Cluster validity indices (CVIs) for evaluating the result of the optimal number of clusters are critical measures in clustering problems. Most CVIs are designed for typical data-type objects called certain data objects. Certain data objects only have a singular value and include no uncertainty, so they are assumed to be information-abundant in the real world. In this study, new CVIs for uncertain data, based on kernel probabilistic distance measures to calculate the distance between two distributions in feature space, are proposed for uncertain clusters with arbitrary shapes, sub-clusters, and noise in objects. By transforming original uncertain data into kernel spaces, the proposed CVI accurately measures the compactness and separability of a cluster for arbitrary cluster shapes and is robust to noise and outliers in a cluster. The proposed CVI was evaluated for diverse types of simulated and real-life uncertain objects, confirming that the proposed validity indexes in feature space outperform the pre-existing ones in the original space.

Seagull optimization-based near-duplicate image detection in large image databases

ABSTRACT Many of the near-duplicate (ND) image detection methods involve a greater number of interest points (IPs) and large dimensions of the feature descriptors requiring huge computations and are unsuitable for large image databases. They may fail to detect NDs if the query and images in the database contain sparse IPs due to low entropy. Besides, the k-means algorithm used for the quantization of visual words may land at a sub-optimal minimum for descriptors because of their distance distribution in feature space. This article presents a new ND image detection method, which uniformly distributes the IPs over low and high entropy regions, reduces the dimension of feature descriptors using discrete wavelet transform (DWT) and employs Seagull Optimization algorithm (SOA) for optimally forming the visual words. It examines proposed method performs on image databases of various sizes and shows that the developed method is more reliable and computationally efficient than the alternatives.

Open-Set Classification for Signal Diagnosis of Machinery Sensor in Industrial Environment

In recent years, the signal diagnosis on devices operating under industrial environment has attracted increasing attention. Most data-driven signal diagnosis methods are based on a closed-set assumption that class sets of training and test data are the same. However, in industrial scenarios, during the running process of the device, the operating environment and condition may change over time, continuing generating data belonging to unknown classes with new characteristics and distribution. The unknown classes usually reflect new modes or faults of the device needed to be captured. They are unavailable in training phase, contradicting the closed-set assumption. Existing methods are inappropriate to this type of open-set classification, requiring to classify known classes and recognize unknown classes. To address this challenging problem, this article proposes a generic open-set signal classification method. First, we apply Fourier transform to convert the sensor signals from time domain to frequency domain, then data in the time and frequency domains are fused. Next, a variational encoder-classifier network is proposed to classify known classes and learn the distribution of feature space to extract robust latent features. Finally, based on extreme value theory and entropy, a pair of discriminators determine whether samples belong to unknown or not. The experimental results on two vibration-signal datasets from bearings and nuclear reactor demonstrate the effectiveness and superiority of our proposed open-set signal classification method, especially in practical applications.

Spectral Characteristics of the Dynamic World Land Cover Classification

The Dynamic World product is a discrete land cover classification of Sentinel 2 reflectance imagery that is global in extent, retrospective to 2015, and updated continuously in near real time. The classifier is trained on a stratified random sample of 20,000 hand-labeled 5 × 5 km Sentinel 2 tiles spanning 14 biomes globally. Since the training data are based on visual interpretation of image composites by both expert and non-expert annotators, without explicit spectral properties specified in the class definitions, the spectral characteristics of the classes are not obvious. The objective of this study is to quantify the physical distinctions among the land cover classes by characterizing the spectral properties of the range of reflectance present within each of the Dynamic World classes over a variety of landscapes. This is achieved by comparing both the eight-class probability feature space (excluding snow) and the maximum probability class assignment (label) distributions to continuous land cover fraction estimates derived from a globally standardized spectral mixture model. Standardized substrate, vegetation, and dark (SVD) endmembers are used to unmix nine Sentinel 2 reflectance tiles from nine spectral diversity hotspots for comparison between the SVD land cover fraction continua and the Dynamic World class probability continua and class assignments. The variance partition for the class probability feature spaces indicates that eight of these nine hotspots are effectively five-dimensional to 95% of variance. Class probability feature spaces of the hotspots all show a tetrahedral form with probability continua spanning multiple classes. Comparison of SVD land cover fraction distributions with maximum probability class assignments (labels) and probability feature space distributions reveal a clear distinction between (1) physically and spectrally heterogeneous biomes characterized by continuous gradations in vegetation density, substrate albedo, and structural shadow fractions, and (2) more homogeneous biomes characterized by closed canopy vegetation (forest) or negligible vegetation cover (e.g., desert, water). Due to the ubiquity of spectrally heterogeneous biomes worldwide, the class probability feature space adds considerable value to the Dynamic World maximum probability class labels by offering users the opportunity to depict inherently gradational heterogeneous landscapes otherwise not generally offered with other discrete thematic classifications.