Representation Learning Ability Research Articles

Curriculumformer: Taming Curriculum Pre-Training for Enhanced 3-D Point Cloud Understanding.

Learning universal representations of 3-D point clouds is essential for reducing the need for manual annotation of large-scale and irregular point cloud datasets. The current modus operandi for representative learning is self-supervised learning, which has shown great potential for improving point cloud understanding. Nevertheless, it remains an open problem how to employ auto-encoding for learning universal 3-D representations of irregularly structured point clouds, as previous methods focus on either global shapes or local geometries. To this end, we present a cascaded self-supervised point cloud representation learning framework, dubbed Curriculumformer, aiming to tame curriculum pre-training for enhanced point cloud understanding. Our main idea lies in devising a progressive pre-training strategy, which trains the Transformer in an easy-to-hard manner. Specifically, we first pre-train the Transformer using an upsampling strategy, which allows it to learn global information. Then, we follow up with a completion strategy, which enables the Transformer to gain insight into local geometries. Finally, we propose a Multi-Modal Multi-Modality Contrastive Learning (M4CL) strategy to enhance the ability of representation learning by enriching the Transformer with semantic information. In this way, the pre-trained Transformer can be easily transferred to a wide range of downstream applications. We demonstrate the superior performance of Curriculumformer on various discriminant and generative tasks, outperforming state-of-the-art methods. Moreover, Curriculumformer can also be integrated into other off-the-shelf methods to promote their performance. Our code is available at https://github.com/Fayeben/Curriculumformer.

Enhancing Anomaly Detection in Structured Data Using Siamese Neural Networks as a Feature Extractor

Anomaly detection in structured data presents significant challenges, particularly in scenarios with extreme class imbalance. The Siamese Neural Network (SNN) is traditionally recognized for its ability to measure pairwise similarities, rather than being utilized as a feature extractor. However, in this study, we introduce a novel approach by leveraging the feature extraction capabilities of SNN, inspired by the powerful representation learning ability of neural networks. We integrate SNN with four different classifiers and the Synthetic Minority Over-sampling Technique (SMOTE) for supervised anomaly detection and evaluate its performance across five structured datasets under varying anomaly ratios. Our findings reveal that, when used as a feature extractor, SNN significantly enhances classification performance and demonstrates superior robustness compared to traditional anomaly detection methods, particularly under extreme class imbalance. These results highlight the potential of repurposing SNN beyond similarity learning, offering a scalable and effective feature extraction framework for anomaly detection in structured data applications.

CA-SQBG: Cross-attention guided Siamese quantum BiGRU for drug-drug interaction extraction.

Accurate and efficient drug-drug interaction extraction (DDIE) from the medical corpus is essential for pharmacovigilance, drug therapy and drug development. To solve the problems of unbalance dataset and lack of accurate manual annotations in DDIE, a cross-attention guided Siamese quantum BiGRU (CA-SQBG) is constructed to improve feature representation learning ability for DDIE. It mainly consists of two quantum BiGRUs (QBiGRUs) and a cross-attention, where two QBiGRUs are Siamese implemented in a variational quantum environment to learn the contextual semantic feature representation of drug pairs, cross-attention is employed to learn mutual information from the Siamese QBiGRUs, which in turn allows the two modules to extract DDI more collaboratively. Unlike BiGRU, Siamese QBiGRUs uses internal and external dependencies in quaternion algebra to map DDI correlations within and between multidimensional features, whereas BiGRU can only capture dependencies within sequences. CA-SQBG is evaluated on the DDIExtraction2013 dataset, and the results demonstrate that it can effectively capture the inter- and intra-dependencies within multimodal features with few parameters, using a small number of training samples, and is superior to the most advanced DDIE methods. CA-SQBG offers potential applications for quantum computing and Siamese networks in the field of DDIE. Code is available on https://github.com/xaycq/CA-SQBG.

The Data Preprocessing Technique in Machine Learning

Data preprocessing has a vital impact on the performance of traditional machine learning. However, with the continuous development of deep learning technology and the powerful representation learning ability of neural networks, deep learning models can easily convert raw data into continuous feature representation and have made remarkable achievements in many downstream tasks. Recently, with the application of deep learning technology in more fields that require robustness and stability, its model bias caused by data quality problems are gradually exposed, which makes the data preprocessing technology regain the attention of researchers. This paper systematically expounds on the primary data preprocessing technology in machine learning and discusses the model bias caused by data deviation and the robustness in the face of attacks. In addition, this paper also illustrates the possibility of a data preprocessing foundation in solving the defects of deep learning foundation based on glove word vector and image classification task based on convolutional neural network. The research in this paper can provide a valuable reference for researchers in related fields.

An object detection-based model for automated screening of stem-cells senescence during drug screening

An object detection-based model for automated screening of stem-cells senescence during drug screening

Knowledge Representation Learning Method Based on Semantic Enhancement of External Information

Background: Knowledge representation learning aims at mapping entity and relational data in knowledge graphs to a low-dimensional space in the form of vectors. The existing work has mainly focused on structured information representation of triples or introducing only one additional kind of information, which has large limitations and reduces the representation efficiency. Objective: This study aims to combine entity description information and textual relationship description information with triadic structure information, and then use the linear mapping method to linearly transform the structure vector and text vector to obtain the joint representation vector. Methods: A knowledge representation learning (DRKRL) model that fuses external information for semantic enhancement is proposed, which combines entity descriptions and textual relations with a triadic structure. For entity descriptions, a vector representation is performed using a bi-directional long- and short-term memory network (Bi-LSTM) model and an attention mechanism. For the textual relations, a convolutional neural network is used to vectorially encode the relations between entities, and then an attention mechanism is used to obtain valuable information as complementary information to the triad. Results: Link prediction and triadic group classification experiments were conducted on the FB15K, FB15K-237, WN18, WN18RR, and NELL-995 datasets. Theoretical analysis and experimental results show that the DRKRL model proposed in this paper has higher accuracy and efficiency compared with existing models. Conclusion: Combining entity description information and textual relationship description information with triadic structure information can make the model have better performance and effectively improve the knowledge representation learning ability.

Alternating nonnegative least squares-incorporated regularized symmetric latent factor analysis for undirected weighted networks

Alternating nonnegative least squares-incorporated regularized symmetric latent factor analysis for undirected weighted networks

Multi-Task Credible Pseudo-Label Learning for Semi-Supervised Crowd Counting.

As a widely used semi-supervised learning strategy, self-training generates pseudo-labels to alleviate the labor-intensive and time-consuming annotation problems in crowd counting while boosting the model performance with limited labeled data and massive unlabeled data. However, the noise in the pseudo-labels of the density maps greatly hinders the performance of semi-supervised crowd counting. Although auxiliary tasks, e.g., binary segmentation, are utilized to help improve the feature representation learning ability, they are isolated from the main task, i.e., density map regression and the multi-task relationships are totally ignored. To address the above issues, we develop a multi-task credible pseudo-label learning (MTCP) framework for crowd counting, consisting of three multi-task branches, i.e., density regression as the main task, and binary segmentation and confidence prediction as the auxiliary tasks. Multi-task learning is conducted on the labeled data by sharing the same feature extractor for all three tasks and taking multi-task relations into account. To reduce epistemic uncertainty, the labeled data are further expanded, by trimming the labeled data according to the predicted confidence map for low-confidence regions, which can be regarded as an effective data augmentation strategy. For unlabeled data, compared with the existing works that only use the pseudo-labels of binary segmentation, we generate credible pseudo-labels of density maps directly, which can reduce the noise in pseudo-labels and therefore decrease aleatoric uncertainty. Extensive comparisons on four crowd-counting datasets demonstrate the superiority of our proposed model over the competing methods. The code is available at: https://github.com/ljq2000/MTCP.

Parallel Adaptive Stochastic Gradient Descent Algorithms for Latent Factor Analysis of High-Dimensional and Incomplete Industrial Data

Latent factor analysis (LFA) is efficient in knowledge discovery from a high-dimensional and incomplete (HDI) matrix frequently encountered in industrial big data-related applications. A stochastic gradient descent (SGD) algorithm is commonly adopted as a learning algorithm for LFA owing to its high efficiency. However, its sequential nature makes it less scalable when processing large-scale data. Although alternating SGD decouples an LFA process to achieve parallelization, its performance relies on its hyper-parameters that are highly expensive to tune. To address this issue, this paper presents three extended alternating SGD algorithms whose hyper-parameters are made adaptive through particle swarm optimization. Correspondingly, three Parallel Adaptive LFA (PAL) models are proposed and achieve highly efficient latent factor acquisition from an HDI matrix. Experiments have been conducted on four HDI matrices collected from industrial applications, and the benchmark models are LFA models based on state-of-the-art parallel SGD algorithms including the alternative SGD, Hogwild!, distributed gradient descent, and sparse matrix factorization parallelization. The results demonstrate that compared with the benchmarks, with 32 threads, the proposed PAL models achieve much speedup gain. They achieve the highest prediction accuracy for missing data on most cases. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —HDI data are commonly encountered in many industrial big data-related applications, where rich knowledge and patterns can be extracted efficiently. An SGD based-LFA model is popular in addressing HDI data due to its efficiency. Yet when dealing with large-scale HDI data, its serial nature greatly reduces its scalability. Although alternating SGD can decouple an LFA process to implement parallelization, its performance depends on its hyper-parameter whose tuning is tedious. To address this vital issue, this study proposes three extended alternating SGD algorithms whose hyper-parameters are made via through a particle swarm optimizer. Based on them, three models are realized, which are able to efficiently obtain latent factors from HDI matrices. Compared with the existing and state-of-the-art models, they enjoy their hyper-parameter-adaptive learning process, as well as highly competitive computational efficiency and representation learning ability. Hence, they provide practitioners with more scalable solutions when addressing large HDI data from industrial applications.

Enhancing generalizability and performance in drug-target interaction identification by integrating pharmacophore and pre-trained models.

In drug discovery, it is crucial to assess the drug-target binding affinity (DTA). Although molecular docking is widely used, computational efficiency limits its application in large-scale virtual screening. Deep learning-based methods learn virtual scoring functions from labeled datasets and can quickly predict affinity. However, there are three limitations. First, existing methods only consider the atom-bond graph or one-dimensional sequence representations of compounds, ignoring the information about functional groups (pharmacophores) with specific biological activities. Second, relying on limited labeled datasets fails to learn comprehensive embedding representations of compounds and proteins, resulting in poor generalization performance in complex scenarios. Third, existing feature fusion methods cannot adequately capture contextual interaction information. Therefore, we propose a novel DTA prediction method named HeteroDTA. Specifically, a multi-view compound feature extraction module is constructed to model the atom-bond graph and pharmacophore graph. The residue concat graph and protein sequence are also utilized to model protein structure and function. Moreover, to enhance the generalization capability and reduce the dependence on task-specific labeled data, pre-trained models are utilized to initialize the atomic features of the compounds and the embedding representations of the protein sequence. A context-aware nonlinear feature fusion method is also proposed to learn interaction patterns between compounds and proteins. Experimental results on public benchmark datasets show that HeteroDTA significantly outperforms existing methods. In addition, HeteroDTA shows excellent generalization performance in cold-start experiments and superiority in the representation learning ability of drug-target pairs. Finally, the effectiveness of HeteroDTA is demonstrated in a real-world drug discovery study. The source code and data are available at https://github.com/daydayupzzl/HeteroDTA.

Graph Aggregating-Repelling Network: Do Not Trust All Neighbors in Heterophilic Graphs

Graph Aggregating-Repelling Network: Do Not Trust All Neighbors in Heterophilic Graphs

GRLC: Graph Representation Learning With Constraints.

Contrastive learning has been successfully applied in unsupervised representation learning. However, the generalization ability of representation learning is limited by the fact that the loss of downstream tasks (e.g., classification) is rarely taken into account while designing contrastive methods. In this article, we propose a new contrastive-based unsupervised graph representation learning (UGRL) framework by 1) maximizing the mutual information (MI) between the semantic information and the structural information of the data and 2) designing three constraints to simultaneously consider the downstream tasks and the representation learning. As a result, our proposed method outputs robust low-dimensional representations. Experimental results on 11 public datasets demonstrate that our proposed method is superior over recent state-of-the-art methods in terms of different downstream tasks. Our code is available at https://github.com/LarryUESTC/GRLC.

HGCLAMIR: Hypergraph contrastive learning with attention mechanism and integrated multi-view representation for predicting miRNA-disease associations.

Existing studies have shown that the abnormal expression of microRNAs (miRNAs) usually leads to the occurrence and development of human diseases. Identifying disease-related miRNAs contributes to studying the pathogenesis of diseases at the molecular level. As traditional biological experiments are time-consuming and expensive, computational methods have been used as an effective complement to infer the potential associations between miRNAs and diseases. However, most of the existing computational methods still face three main challenges: (i) learning of high-order relations; (ii) insufficient representation learning ability; (iii) importance learning and integration of multi-view embedding representation. To this end, we developed a HyperGraph Contrastive Learning with view-aware Attention Mechanism and Integrated multi-view Representation (HGCLAMIR) model to discover potential miRNA-disease associations. First, hypergraph convolutional network (HGCN) was utilized to capture high-order complex relations from hypergraphs related to miRNAs and diseases. Then, we combined HGCN with contrastive learning to improve and enhance the embedded representation learning ability of HGCN. Moreover, we introduced view-aware attention mechanism to adaptively weight the embedded representations of different views, thereby obtaining the importance of multi-view latent representations. Next, we innovatively proposed integrated representation learning to integrate the embedded representation information of multiple views for obtaining more reasonable embedding information. Finally, the integrated representation information was fed into a neural network-based matrix completion method to perform miRNA-disease association prediction. Experimental results on the cross-validation set and independent test set indicated that HGCLAMIR can achieve better prediction performance than other baseline models. Furthermore, the results of case studies and enrichment analysis further demonstrated the accuracy of HGCLAMIR and unconfirmed potential associations had biological significance.

Optimization of financial data processing strategy based on LSTM-GCN

In the process of training LSTM-GCN mixed model with labeled data, a common practice is to divide the data set into a training set and a validation set, and cross-validation techniques are frequently used to ensure the effective partitioning of data. The training phase mainly involves the selection of optimization methods such as gradient descent to update the model parameters, so that the model can correctly identify the feature patterns in the data. In this process, we first define a loss function to quantify the difference between the model's predicted value and the actual value. For example, when dealing with classification problems, the cross-entropy loss function is often used; Whereas in regression problems, the mean square error loss function or the absolute error loss function is more common. Then, the loss gradient of each parameter is calculated by backpropagation algorithm, and optimization strategies such as gradient descent are applied to adjust the parameter values. Through repeated iterations, these algorithms aim to minimize the loss function as a core goal for training the model. In addition, the details of model training, such as setting the appropriate learning rate, taking the appropriate regularization measures, and choosing the appropriate weight initialization strategy, are also key steps that cannot be ignored. This paper conforms to the research and development trend of the industry, focusing on how to use the LSTM-GCN model to capture temporal and spatial characteristics, process multi-modal data, process nonlinear data and dynamic weight allocation, etc., providing new ideas for Internet finance companies in data processing. This model combines the advantages of LSTM and GCN, and has prominent advantages such as timeliness, strong representation learning ability, associability, and interpretability. It can process both time series data and graphic data, and can analyze and mine the correlation and regularity in large-scale time series data, so as to process financial data more accurately. It greatly reduces the risk of investment decision.

Symmetry and Graph Bi-Regularized Non-Negative Matrix Factorization for Precise Community Detection

Community is a fundamental and highly desired pattern in a Large-scale Undirected Network (LUN). Community detection is a vital issue when LUN representation learning is performed. Owing to its good scalability and interpretability, a Symmetric and Non-negative Matrix Factorization model is frequently utilized to tackle this issue. It adopts a unique Latent Factor (LF) matrix for precisely representing LUN’s symmetry, which, unfortunately, leads to a reduced LF space that decreases its representation learning ability to a target LUN. Motivated by this discovery, this study proposes a Symmetry and Graph Bi-regularized Non-negative Matrix Factorization (B-NMF) method that: a) leverages multiple LF matrices when representing LUN, thereby boosting the representation learning ability; b) constructs a symmetry regularization term that implies the equality constraint among its multiple LF matrices, thereby illustrating LUN’s intrinsic symmetry; and c) incorporates graph regularization into its learning objective, thereby illustrating LUN’s local geometry. A theoretical proof is given to theoretically validate B-NMF’s convergence ability. The regularization hyperparameters are selected by validating model modularity, thereby guaranteeing B-NMF’s practicability in addressing real application issues. Extensive experimental results on ten LUNs from real applications demonstrate that the proposed B-NMF-based community detector significantly outperforms several baseline and state-of-the-art models in achieving highly-accurate community detection results. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —LUNs are very-commonly seen in real applications like a social network system. Communities in LUNs are vital for various knowledge discovery-related applications. For accurately detecting them, a detector should guarantee its high representation learning ability to a target LUN. To do so, this paper presents a B-NMF model that is able to perform precise representation learning to LUNs, thereby achieving accurate community detection results. In comparison with conventional Symmetric and Non-negative Matrix Factorization-based community detectors, a B-NMF-based community detector enjoys its enlarged latent feature space, which ensures its higher representation ability to a target LUN. It depends on two regularization hyperparameters, which can be selected by performing grid-search on the target LUN via its modularity evaluation. This paper gives the empirical values of B-NMF’s regularization hyperparameters based on the parametersensitivity tests on the involved experimental datasets. The proposed B-NMF model is shown to be highly suitable for addressing community detection and clustering tasks on LUNs from real applications.

A novel method for detecting credit card fraud problems.

Credit card fraud is a significant problem that costs billions of dollars annually. Detecting fraudulent transactions is challenging due to the imbalance in class distribution, where the majority of transactions are legitimate. While pre-processing techniques such as oversampling of minority classes are commonly used to address this issue, they often generate unrealistic or overgeneralized samples. This paper proposes a method called autoencoder with probabilistic xgboost based on SMOTE and CGAN(AE-XGB-SMOTE-CGAN) for detecting credit card frauds.AE-XGB-SMOTE-CGAN is a novel method proposed for credit card fraud detection problems. The credit card fraud dataset comes from a real dataset anonymized by a bank and is highly imbalanced, with normal data far greater than fraud data. Autoencoder (AE) is used to extract relevant features from the dataset, enhancing the ability of feature representation learning, and are then fed into xgboost for classification according to the threshold. Additionally, in this study, we propose a novel approach that hybridizes Generative Adversarial Network (GAN) and Synthetic Minority Over-Sampling Technique (SMOTE) to tackle class imbalance problems. Our two-phase oversampling approach involves knowledge transfer and leverages the synergies of SMOTE and GAN. Specifically, GAN transforms the unrealistic or overgeneralized samples generated by SMOTE into realistic data distributions where there is not enough minority class data available for GAN to process effectively on its own. SMOTE is used to address class imbalance issues and CGAN is used to generate new, realistic data to supplement the original dataset. The AE-XGB-SMOTE-CGAN algorithm is also compared to other commonly used machine learning algorithms, such as KNN and Light GBM, and shows an overall improvement of 2% in terms of the ACC index compared to these algorithms. The AE-XGB-SMOTE-CGAN algorithm also outperforms KNN in terms of the MCC index by 30% when the threshold is set to 0.35. This indicates that the AE-XGB-SMOTE-CGAN algorithm has higher accuracy, true positive rate, true negative rate, and Matthew's correlation coefficient, making it a promising method for detecting credit card fraud.

A Prediction-Sampling-Based Multilayer-Structured Latent Factor Model for Accurate Representation to High-Dimensional and Sparse Data.

Performing highly accurate representation learning on a high-dimensional and sparse (HiDS) matrix is of great significance in a big data-related application such as a recommender system. A latent factor (LF) model is one of the most efficient approaches to the HiDS matrix representation. However, an LF model's representation learning ability relies heavily on an HiDS matrix's known data density, which is extremely low due to numerous missing data entities. To address this issue, this work proposes a prediction-sampling-based multilayer-structured LF (PMLF) model with twofold ideas: 1) constructing a loosely connected multilayered LF architecture to increase the known data density of an input HiDS matrix by generating synthetic data layer by layer and 2) constraining this synthetic data generating process through a random prediction-sampling strategy and nonlinear activations to avoid overfitting. In the experiments, PMLF is compared with six state-of-the-art LF-and deep neural network (DNN)-based models on four HiDS matrices from industrial applications. The results demonstrate that PMLF outperforms its peers in well-balancing prediction accuracy and computational efficiency.

Multi-scale residual neural network with enhanced gated recurrent unit for fault diagnosis of rolling bearing

As the key component of rotating machinery, effective and reliable fault diagnosis of rolling bearing is particularly critical for promoting production safety and economic benefits. The powerful representation learning ability of convolutional neural network (CNN) enables it to effectively extract fault information from vibration signals of rolling bearing. Nevertheless, challenges are faced by CNN in extracting features at multi-scale and capturing temporal features. With regard to this issue, a hybrid deep learning model that incorporates the multi-scale residual neural network (MSRN) with the enhanced gated recurrent unit (EGRU), namely MSRN-EGRU, is proposed in this paper. To begin with, MSRN is designed by introducing multi-scale structure and residual connections into CNN for extracting local features effectively and improving the feature representation of the model. Then, the extracted local features are input into EGRU to further extract temporal features, where EGRU is proposed by improving GRU structure and embedding scaled exponential liner unit (SELU), which enhances the nonlinear modeling and memory ability. Eventually, the obtained features are processed by α-Dropout and global average pooling before being inputted into the softmax layer for fault diagnosis. To validate the effectiveness of the proposed model, three baseline models and two ablation models were employed for comparative experiments with two bearing datasets. The experimental results reveal that the proposed model achieves commendable performance in terms of accuracy, robustness, and convergence for fault diagnosis of rolling bearing.

NanoCon: contrastive learning-based deep hybrid network for nanopore methylation detection.

5-Methylcytosine (5mC), a fundamental element of DNA methylation in eukaryotes, plays a vital role in gene expression regulation, embryonic development, and other biological processes. Although several computational methods have been proposed for detecting the base modifications in DNA like 5mC sites from Nanopore sequencing data, they face challenges including sensitivity to noise, and ignoring the imbalanced distribution of methylation sites in real-world scenarios. Here, we develop NanoCon, a deep hybrid network coupled with contrastive learning strategy to detect 5mC methylation sites from Nanopore reads. In particular, we adopted a contrastive learning module to alleviate the issues caused by imbalanced data distribution in nanopore sequencing, offering a more accurate and robust detection of 5mC sites. Evaluation results demonstrate that NanoCon outperforms existing methods, highlighting its potential as a valuable tool in genomic sequencing and methylation prediction. In addition, we also verified the effectiveness of our representation learning ability on two datasets by visualizing the dimension reduction of the features of methylation and nonmethylation sites from our NanoCon. Furthermore, cross-species and cross-5mC methylation motifs experiments indicated the robustness and the ability to perform transfer learning of our model. We hope this work can contribute to the community by providing a powerful and reliable solution for 5mC site detection in genomic studies. The project code is available at https://github.com/Challis-yin/NanoCon.

A deep latent space model for interpretable representation learning on directed graphs

A deep latent space model for interpretable representation learning on directed graphs