Data Representation Learning Research Articles

Quantum-Inspired Data Embedding for Unlabeled Data in Sparse Environments: A Theoretical Framework for Improved Semi-Supervised Learning without Hardware Dependence

This paper introduces an innovative theoretical framework for quantum-inspired data embeddings, grounded in foundational concepts of quantum mechanics such as superposition and entanglement. This approach aims to advance semi-supervised learning in contexts characterized by limited labeled data by enabling more intricate and expressive embeddings that capture the underlying structure of the data effectively. Grounded in foundational quantum mechanics concepts such as superposition and entanglement, this approach redefines data representation by enabling more intricate and expressive embeddings. Emulating quantum superposition encodes each data point as a probabilistic amalgamation of multiple feature states, facilitating a richer, multidimensional representation of underlying structures and patterns. Additionally, quantum-inspired entanglement mechanisms are harnessed to model intricate dependencies between labeled and unlabeled data, promoting enhanced knowledge transfer and structural inference within the learning paradigm. In contrast to conventional quantum machine learning methodologies that often rely on quantum hardware, this framework is fully realizable within classical computational architectures, thus bypassing the practical limitations of quantum hardware. The versatility of this model is illustrated through its application to critical domains such as medical diagnosis, resource-constrained natural language processing, and financial forecasting—areas where data scarcity impedes the efficacy of traditional models. Experimental evaluations reveal that quantum-inspired embeddings substantially outperform standard approaches, enhancing model resilience and generalization in high-dimensional, low-sample scenarios. This research marks a significant stride in integrating quantum theoretical principles with classical machine learning, broadening the scope of data representation and semi-supervised learning while circumventing the technological barriers of quantum computing infrastructure.

Patch-Wise-Based Self-Supervised Learning for Anomaly Detection on Multivariate Time Series Data

Multivariate time series anomaly detection is a crucial technology to prevent unexpected errors from causing critical impacts. Effective anomaly detection in such data requires accurately capturing temporal patterns and ensuring the availability of adequate data. This study proposes a patch-wise framework for anomaly detection. The proposed approach comprises four key components: (i) maintaining continuous features through patching, (ii) incorporating various temporal information by learning channel dependencies and adding relative positional bias, (iii) achieving feature representation learning through self-supervised learning, and (iv) supervised learning based on anomaly augmentation for downstream tasks. The proposed method demonstrates strong anomaly detection performance by leveraging patching to maintain temporal continuity while effectively learning data representations and handling downstream tasks. Additionally, it mitigates the issue of insufficient anomaly data by supporting the learning of diverse types of anomalies. The experimental results show that our model achieved a 23% to 205% improvement in the F1 score compared to existing methods on datasets such as MSL, which has a relatively small amount of training data. Furthermore, the model also delivered a competitive performance on the SMAP dataset. By systematically learning both local and global dependencies, the proposed method strikes an effective balance between feature representation and anomaly detection accuracy, making it a valuable tool for real-world multivariate time series applications.

GAADE: identification spatially variable genes based on adaptive graph attention network.

The rapid advancement of spatial transcriptomics (ST) sequencing technology has made it possible to capture gene expression with spatial coordinate information at the cellular level. Although many methods in ST data analysis can detect spatially variable genes (SVGs), these methods often fail to identify genes with explicit spatial expression patterns due to the lack of consideration for spatial domains. Considering spatial domains is crucial for identifying SVGs as it focuses the analysis of gene expression changes on biologically relevant regions, aiding in the more accurate identification of SVGs associated with specific cell types. Existing methods for identifying SVGs based on spatial domains predefine spot similarity before training, which prevents adaptive learning and limits generalizability across different tissues or samples. This limitation may also lead to inaccurate identification of specific genes at boundary regions. To address these issues, we present GAADE, an unsupervised neural network architecture based on graph-structured data representation learning. GAADE stacks encoder/decoder layers and integrates a self-attention mechanism to reconstruct node attributes and graph structure, effectively capturing spatial domain structures of different sections. Consequently, we confine the identification of SVGs within spatial domains. By performing differential expression analysis on spots within the target spatial domain and their multi-order neighbors, GAADE detects genes with enriched expression patterns within defined domains. Comparative evaluations with five other popular methods on ST datasets across four different species, regions and tissues demonstrate that GAADE exhibits superior performance in detecting SVGs and capturing the extent of spatial gene expression variation.

Research on positive and negative sample sampling strategies in contrastive learning

Abstract. Contrastive learning is an important technique in the field of machine learning for learning data representations. In the field of self-supervised visual representation learning, the strategy of positive and negative sample selection is key to improving the efficiency and effectiveness of model learning. Traditional self-supervised learning methods often employ random sampling strategies to select positive and negative samples, but this approach may result in uneven sample quality when dealing with complex datasets, thereby affecting the learning outcomes. To alleviate this problem, this study is dedicated to exploring more effective strategies for positive and negative sample selection and processing to optimize the self-supervised learning process. To this end, we propose an improved method of self-supervised learning called contrastive learning with enhanced diversity. On the one hand, this method utilizes the weight parameters of the DINO pre-trained model to initialize the feature extraction network of SimCLR, providing more accurate calculations of feature similarity. On the other hand, by setting a threshold on the feature similarity matrix and penalizing (subtracting 0.5 from) similarity scores that do not exceed this threshold, we reduce the excessive impact of high similarity scores on model training, thereby helping the model better distinguish between positive and negative samples. In downstream image classification tasks, we conducted detailed evaluations of the improved model, specifically including fine-tuning and linear evaluation aspects. Experiments show that the proposed approach improves the performance of the loss functions and improves the accuracy of the proposed SimCLR model.

Latent Representation Learning for Geospatial Entities

Representation learning has been instrumental in the success of machine learning, offering compact and performant data representations for diverse downstream tasks. In the spatial domain, it has been pivotal in extracting latent patterns from various data types, including points, polylines, polygons, and networked structures. However, existing approaches often fall short of explicitly capturing both semantic and spatial information, relying on proxies and synthetic features. This article presents GeoNN, a novel graph neural network-based model designed to learn spatially-aware embeddings for geospatial entities. GeoNN leverages edge features generated from geodesic functions, dynamically selecting relevant features based on relative locations. It introduces both transductive (GeoNN-T) and inductive (GeoNN-I) models, ensuring effective encoding of geospatial features and scalability with entity changes. Extensive experiments demonstrate GeoNN's effectiveness in location-sensitive superpixel-based graphs and real-world points of interest, outperforming baselines across various evaluation measures.

Joint Image Processing with Learning-Driven Data Representation and Model Behavior for Non-Intrusive Anemia Diagnosis in Pediatric Patients.

Anemia diagnosis is crucial for pediatric patients due to its impact on growth and development. Traditional methods, like blood tests, are effective but pose challenges, such as discomfort, infection risk, and frequent monitoring difficulties, underscoring the need for non-intrusive diagnostic methods. In light of this, this study proposes a novel method that combines image processing with learning-driven data representation and model behavior for non-intrusive anemia diagnosis in pediatric patients. The contributions of this study are threefold. First, it uses an image-processing pipeline to extract 181 features from 13 categories, with a feature-selection process identifying the most crucial data for learning. Second, a deep multilayered network based on long short-term memory (LSTM) is utilized to train a model for classifying images into anemic and non-anemic cases, where hyperparameters are optimized using Bayesian approaches. Third, the trained LSTM model is integrated as a layer into a learning model developed based on recurrent expansion rules, forming a part of a new deep network called a recurrent expansion network (RexNet). RexNet is designed to learn data representations akin to traditional deep-learning methods while also understanding the interaction between dependent and independent variables. The proposed approach is applied to three public datasets, namely conjunctival eye images, palmar images, and fingernail images of children aged up to 6 years. RexNet achieves an overall evaluation of 99.83 ± 0.02% across all classification metrics, demonstrating significant improvements in diagnostic results and generalization compared to LSTM networks and existing methods. This highlights RexNet's potential as a promising alternative to traditional blood-based methods for non-intrusive anemia diagnosis.

Enhancing capabilities of generative models through VAE-GAN integration: A review

Our review explores the integration of Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs), which are pivotal in the realm of generative models. VAEs are renowned for their robust probabilistic foundations and capacity for complex data representation learning, while GANs are celebrated for generating high-fidelity images. Despite their strengths, both models have limitations: VAEs often produce less sharp outputs, and GANs face challenges with training stability. The hybrid VAE-GAN models harness the strengths of both architectures to overcome these limitations, enhancing output quality and diversity. We provide a comprehensive overview of VAEs and GANs technology developments, their integration strategies, and resultant performance improvements. Applications across various fields, such as artistic creation, medical imaging, e-commerce, and video gaming, highlight the transformative potential of these models. However, challenges in model robustness, ethical concerns, and computational demands persist, posing significant hurdles. Future research directions are poised to transform the VAE-GAN landscape significantly. Enhancing training stability remains a priority, with new approaches such as incorporating self-correcting mechanisms into GANs training being tested. Addressing ethical issues is also critical, as policymakers and technologists work together to develop standards that prevent misuse. Moreover, reducing computational costs is fundamental to democratizing access to these technologies. Projects such as the development of MobileNetV2 have made strides in creating more efficient neural network architectures that maintain performance while being less resource-intensive. Further, the exploration of VAE-GAN applications in fields like augmented reality and personalized medicine offers exciting opportunities for growth, as evidenced by recent pilot studies.

Enhanced Network Intrusion Detection System for Internet of Things Security Using Multimodal Big Data Representation with Transfer Learning and Game Theory.

Internet of Things (IoT) applications and resources are highly vulnerable to flood attacks, including Distributed Denial of Service (DDoS) attacks. These attacks overwhelm the targeted device with numerous network packets, making its resources inaccessible to authorized users. Such attacks may comprise attack references, attack types, sub-categories, host information, malicious scripts, etc. These details assist security professionals in identifying weaknesses, tailoring defense measures, and responding rapidly to possible threats, thereby improving the overall security posture of IoT devices. Developing an intelligent Intrusion Detection System (IDS) is highly complex due to its numerous network features. This study presents an improved IDS for IoT security that employs multimodal big data representation and transfer learning. First, the Packet Capture (PCAP) files are crawled to retrieve the necessary attacks and bytes. Second, Spark-based big data optimization algorithms handle huge volumes of data. Second, a transfer learning approach such as word2vec retrieves semantically-based observed features. Third, an algorithm is developed to convert network bytes into images, and texture features are extracted by configuring an attention-based Residual Network (ResNet). Finally, the trained text and texture features are combined and used as multimodal features to classify various attacks. The proposed method is thoroughly evaluated on three widely used IoT-based datasets: CIC-IoT 2022, CIC-IoT 2023, and Edge-IIoT. The proposed method achieves excellent classification performance, with an accuracy of 98.2%. In addition, we present a game theory-based process to validate the proposed approach formally.

Review: Deep Learning and Fuzzy Logic Applications

The modeling and prediction field bosses a variety of practical applications, deep learning is a powerful tool used in this field. It has been proved that deep learning is a useful technique for extracting extremely accurate predictions from complex data sources, and also Recursive neural networks have demonstrated their usefulness in language translation and caption production, but convolutional neural networks remain the dominant solution for image classification tasks. In addition, deep learning, also known as deep neural networks, involves training models with multiple layers of interconnected artificial neurons. The primary idea of deep learning is to learn data representations through raising levels of abstraction. These strategies are effective, but they don't explain how the result is produced. Without knowing how a solution is arrived at using deep learning. In the field of artificial intelligence, deep learning and fuzzy logic are two powerful techniques. In addition, fuzzy logic combines deep learning will help deep learning select the desired features and work without supervision, this will make it possible to develop reliable systems with rich DL information even in the absence of hand-labeled data. Fuzzy logic that interpreted these features will subsequently provide explanations for the system's choice of classification label. This survey highlights the various applications which use fuzzy logic to improve deep learning.

Cross-layer self-representation enhanced deep subspace clustering with self-supervision

Deep subspace clustering typically employs a self-representation layer to accurately capture the similarity relationships among data points. While current techniques enhance self-representation matrices through optimized network structures and balanced regularization strategies, they often overlook comprehensive learning of global data characteristics. Moreover, scalability to large datasets is limited due to the proportional growth of the self-representation matrix with dataset size, presenting computational challenges. To address these issues, we introduce the Cross-Layer Self-Representation Enhanced Deep Subspace Clustering with Self-Supervision method. This novel approach features a hierarchical self-representation fusion mechanism that enriches the understanding of data relationships across different layers. In addition, we employ a contrastive learning strategy to refine data representation learning further. Crucially, we develop a joint framework that surmounts previous limitations in processing large-scale data. Our method's effectiveness and superiority are conclusively demonstrated through extensive experimental validation.

CDRM: Causal disentangled representation learning for missing data

Missing data pose significant challenges during representation learning of observational data. The incompleteness of data can result in a deterioration of generative performance in disentangled representation learning. Conventional data imputation solutions, such as regression imputation or multiple imputation, often neglect the underlying causal relationships among the data. To address these issues, the causal disentangled representation learning for missing data (CDRM) framework was proposed. Missing data with graph representations are integrated to construct heterogeneous networks composed of observations, features, and known feature values. To achieve data completeness, an interaction module consisting of a parallel neighbor interaction layer and an embedded update layer are integrated with the heterogeneous network to predict missing values. To recover the true causal relationship of missing data, edge embeddings are further introduced during message passing of heterogeneous networks, which can capture the interaction of different features and enrich the representation of observations. Furthermore, the causal relationship is incorporated into the VAE using two distinct encoders to learn representations of causally related concepts. In a series of experimental evaluations on diverse datasets, CDRM consistently outperforms the state-of-the-art method, namely, CausalVAE, in disentangled representation learning, particularly in scenarios with limited labeled data. Notably, CDRM can generate counterfactual data in response to missing data, further enhancing its utility in machine learning applications. The source code and data are available at https://github.com/Causal-Disentangled/CDRM.

Enhancing Time Series Predictability via Structure‐Aware Reservoir Computing

Accurate prediction of the future evolution of observational time series is a paramount challenge in current data‐driven research. While existing techniques struggle to learn useful representations from the temporal correlations, the high dimensionality in spatial domain is always considered as obstacle, leading to the curse of dimensionality and excessive resource consumption. This work designs a novel structure‐aware reservoir computing aiming at enhancing the predictability of coupled time series, by incorporating their historical dynamics as well as structural information. Paralleled reservoir computers with redesigned mixing inputs based on spatial relationships are implemented to cope with the multiple time series, whose core idea originates from the principle of the celebrated Granger causality. Representative numerical simulations and comparisons demonstrate the superior performance of the approach over the traditional ones. This work provides valuable insights into deeply mining both temporal and spatial information to enhance the representation learning of data in various machine learning techniques.

Towards Improved Privacy in Digital Marketing: A Unified Approach to User Modeling with Deep Learning on a Data Monetization Platform

This paper introduces an innovative method for safeguarding user privacy in digital marketing campaigns through the application of deep learning techniques on a data monetization platform. This framework empowers users to maintain authority over their personal data while enabling marketers to pinpoint suitable target audiences. The system consists of several key stages Data representation learning in hyperbolic space captures latent user interests across various data sources with hierarchical structures. Subsequently, Generative Adversarial Networks generate synthetic user interests from these embedding. To preserve user privacy, Federated Learning is utilized for decentralized user monetization, Data privacy, modeling training, ensuring data remains undisclosed to marketers. Lastly, a hyperbolic embedding, Federated learning targeting strategy, rooted in recommendation systems, utilizes learned user interests to identify optimal target audiences for digital marketing campaigns. In sum, this approach offers a comprehensive solution for privacy-preserving user modeling in digital marketing.

Factor annealing decoupling compositional training method for imbalanced hyperspectral image classification

AbstractDue to differences in the quantity and size of observed targets, hyperspectral images are characterized by class imbalance. The standard deep learning classification model training scheme optimizes the overall classification error, which may lead to performance imbalance between classes in hyperspectral image classification frameworks. Therefore, a novel factor annealing decoupling compositional training method is proposed in this paper. Without requiring resampling or reweighting, it implicitly modulates the training process, so standard models can sufficiently learn the representation of the minority classes and further be trained as robust classifiers. Specifically, the label‐distribution‐aware margin loss is combined with the error‐rate‐based cross‐entropy loss via combination factor, which considers both imbalanced data representation learning and classifier overall performance. Then, a factor annealing optimization training scheme is designed to adjust the combination factor, which solves the stage division problem of two‐stage decoupling learning. Experimental results on two hyperspectral image datasets demonstrate that, as compared with other competing approaches, the proposed method can continuously and stably optimize the model parameters, achieving improvements in class average metrics and difficult classes without affecting overall classification performance.

Data Representation for Deep Learning - Based Arabic Text Summarization Performance Using Python Results

A sequence-to-sequence model is used as the foundation for a suggested abstractive Arabic text summarizing system. Our goal is to create a sequence-to-sequence model by utilizing multiple deep artificial neural networks and determining which one performs the best. The encoder and decoder have been developed using several layers of recurrent neural networks, gated recurrent units, recursive neural networks, convolutional neural networks, long short-term memory, and bidirectional long short-term memory. We are re-implementing the fundamental summarization model in this study, which uses the sequence-to-sequence framework. Using a Google Colab Jupiter notebook that runs smoothly, we have constructed these models using the Keras library. The results further demonstrate that one of the key techniques that has led to breakthrough performance with deep neural networks is the use of Gensim for word embeddings over other text representations by abstractive summarization models, along with FastText, a library for efficient learning of word representations and sentence classification.

Generation-based Multi-view Contrast for Self-supervised Graph Representation Learning

Graph contrastive learning has made remarkable achievements in the self-supervised representation learning of graph-structured data. By employing perturbation function (i.e., perturbation on the nodes or edges of graph), most graph contrastive learning methods construct contrastive samples on the original graph. However, the perturbation-based data augmentation methods randomly change the inherent information (e.g., attributes or structures) of the graph. Therefore, after nodes embedding on the perturbed graph, we cannot guarantee the validity of the contrastive samples as well as the learned performance of graph contrastive learning. To this end, in this article, we propose a novel generation-based multi-view contrastive learning framework (GMVC) for self-supervised graph representation learning, which generates the contrastive samples based on our generator rather than perturbation function. Specifically, after nodes embedding on the original graph we first employ random walk in the neighborhood to develop multiple relevant node sequences for each anchor node. We then utilize the transformer to generate the representations of relevant contrastive samples of anchor node based on the features and structures of the sampled node sequences. Finally, by maximizing the consistency between the anchor view and the generated views, we force the model to effectively encode graph information into nodes embeddings. We perform extensive experiments of node classification and link prediction tasks on eight benchmark datasets, which verify the effectiveness of our generation-based multi-view graph contrastive learning method.

Dual auto-weighted multi-view clustering via autoencoder-like nonnegative matrix factorization

Multi-view clustering (MVC) can exploit the complementary information among multi-view data to achieve the satisfactory performance, thus having extensive potentials for practical applications. Although Nonnegative Matrix Factorization (NMF) has emerged as an effective technique for MVC, the existing NMF-based methods still have two main limitations: 1) They solely focus on the reconstruction of original data, which can be regarded as the decoder of an autoencoder, while neglecting the low-dimensional representation learning. 2) They lack the ability to effectively capture both linear and nonlinear structures of data. To solve these problems, in this paper, we propose a Dual Auto-weighted multi-view clustering model based on Autoencoder-like NMF (DA2NMF), which enables a comprehensive exploration of both linear and nonlinear structures. Specifically, we establish an autoencoder-like NMF model that learns linear low-dimensional representations by integrating data reconstruction and representation learning within a unified framework. Moreover, the adaptive graph learning is introduced to explore the nonlinear structures in data. We further design a dual auto-weighted strategy to adaptively compute weights for different views and low-dimensional representations, thereby obtaining an enhanced consistent graph. An effective algorithm based on Multiplicative Update Rule (MUR) is developed to solve the DA2NMF with the theoretical convergence guarantee. Experimental results show that the proposed DA2NMF can effectively improve the clustering performance compared with the state-of-the-art MVC algorithms.

Pneumonia prediction on chest x-ray images using deep learning approach

<span lang="EN-US">Coronavirus disease 2019 (COVID-19) is an infectious disease with first symptoms similar to the flu. In many cases, this disease causes pneumonia. Since pulmonary infections can be observed through radiography images, this paper investigates deep learning methods for automatically analyzing query chest x-ray images. In deep learning, computers can automatically identify useful features for the model, directly from the raw data, bypassing the difficult step of manual information refinement. The main part of the deep learning method is the focus on automatically learning data representations. Visual geometry group 16 (VGG16) and DenseNet121 are methods in deep learning. The data used is a chest x-ray of pneumonia. Data is divided into training, testing, and validation. The best method for this research case is VGG16 with 93% accuracy training and 90% accuracy testing. In this study, DenseNet121 obtained accuracy below VGG16, with 92% accuracy in training and 88% for accuracy testing. Parameters have a significant influence on the accuracy of each model, and with the parameters that have been used, the VGG16 is a method that has high accuracy and can be used to predict chest x-ray images aimed at checking pneumonia in patients.</span>

Usage-aware representation learning for critical information identification in transportation networks

Extracting meaningful information from noisy high-dimensional data is attracting increasing attention as richer and higher resolution data is being collected and used for transportation system planning and management purposes. Discovering critical information via effective data representation learning not only helps reduce data dimension, it also enables a deeper understanding of the underlying properties of noisy data, which could then lead to better planning and operations decisions. In this study, we present a new perspective that, unlike most existing approaches in the general data science literature, the design of data representation should go beyond the data itself; it should incorporate an understanding of how the data is used in the domain-specific applications. We further argue that this design philosophy is particularly important for transportation data because of the high spatial correlations of transportation data brought by network interdependence. We propose a usage-aware representation learning framework by incorporating the information loss for downstream application into the data encoding-decoding process. The proposed approach is formulated as a Stiefel manifold optimization problem. The effectiveness of the proposed framework is demonstrated in two network applications: modeling transportation network flows and estimating network-level vehicular emissions. The performance of the learned representation from our approach is compared with existing approaches using multiple evaluation context, including data reconstruction quality, clustering, anomaly detection, and critical information identification, through case studies implemented in Sioux Falls, Boston, and San Jose networks. The good performance of our approach consistently observed in those experiments indicates the importance of incorporating the downstream data usage in the process of data representation learning.

Adaptively-Accelerated Parallel Stochastic Gradient Descent for High-Dimensional and Incomplete Data Representation Learning

Adaptively-Accelerated Parallel Stochastic Gradient Descent for High-Dimensional and Incomplete Data Representation Learning