Data Representation Learning Research Articles

Joint Image Processing with Learning-Driven Data Representation and Model Behavior for Non-Intrusive Anemia Diagnosis in Pediatric Patients.

Anemia diagnosis is crucial for pediatric patients due to its impact on growth and development. Traditional methods, like blood tests, are effective but pose challenges, such as discomfort, infection risk, and frequent monitoring difficulties, underscoring the need for non-intrusive diagnostic methods. In light of this, this study proposes a novel method that combines image processing with learning-driven data representation and model behavior for non-intrusive anemia diagnosis in pediatric patients. The contributions of this study are threefold. First, it uses an image-processing pipeline to extract 181 features from 13 categories, with a feature-selection process identifying the most crucial data for learning. Second, a deep multilayered network based on long short-term memory (LSTM) is utilized to train a model for classifying images into anemic and non-anemic cases, where hyperparameters are optimized using Bayesian approaches. Third, the trained LSTM model is integrated as a layer into a learning model developed based on recurrent expansion rules, forming a part of a new deep network called a recurrent expansion network (RexNet). RexNet is designed to learn data representations akin to traditional deep-learning methods while also understanding the interaction between dependent and independent variables. The proposed approach is applied to three public datasets, namely conjunctival eye images, palmar images, and fingernail images of children aged up to 6 years. RexNet achieves an overall evaluation of 99.83 ± 0.02% across all classification metrics, demonstrating significant improvements in diagnostic results and generalization compared to LSTM networks and existing methods. This highlights RexNet's potential as a promising alternative to traditional blood-based methods for non-intrusive anemia diagnosis.

Enhancing capabilities of generative models through VAE-GAN integration: A review

Our review explores the integration of Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs), which are pivotal in the realm of generative models. VAEs are renowned for their robust probabilistic foundations and capacity for complex data representation learning, while GANs are celebrated for generating high-fidelity images. Despite their strengths, both models have limitations: VAEs often produce less sharp outputs, and GANs face challenges with training stability. The hybrid VAE-GAN models harness the strengths of both architectures to overcome these limitations, enhancing output quality and diversity. We provide a comprehensive overview of VAEs and GANs technology developments, their integration strategies, and resultant performance improvements. Applications across various fields, such as artistic creation, medical imaging, e-commerce, and video gaming, highlight the transformative potential of these models. However, challenges in model robustness, ethical concerns, and computational demands persist, posing significant hurdles. Future research directions are poised to transform the VAE-GAN landscape significantly. Enhancing training stability remains a priority, with new approaches such as incorporating self-correcting mechanisms into GANs training being tested. Addressing ethical issues is also critical, as policymakers and technologists work together to develop standards that prevent misuse. Moreover, reducing computational costs is fundamental to democratizing access to these technologies. Projects such as the development of MobileNetV2 have made strides in creating more efficient neural network architectures that maintain performance while being less resource-intensive. Further, the exploration of VAE-GAN applications in fields like augmented reality and personalized medicine offers exciting opportunities for growth, as evidenced by recent pilot studies.

Enhanced Network Intrusion Detection System for Internet of Things Security Using Multimodal Big Data Representation with Transfer Learning and Game Theory.

Internet of Things (IoT) applications and resources are highly vulnerable to flood attacks, including Distributed Denial of Service (DDoS) attacks. These attacks overwhelm the targeted device with numerous network packets, making its resources inaccessible to authorized users. Such attacks may comprise attack references, attack types, sub-categories, host information, malicious scripts, etc. These details assist security professionals in identifying weaknesses, tailoring defense measures, and responding rapidly to possible threats, thereby improving the overall security posture of IoT devices. Developing an intelligent Intrusion Detection System (IDS) is highly complex due to its numerous network features. This study presents an improved IDS for IoT security that employs multimodal big data representation and transfer learning. First, the Packet Capture (PCAP) files are crawled to retrieve the necessary attacks and bytes. Second, Spark-based big data optimization algorithms handle huge volumes of data. Second, a transfer learning approach such as word2vec retrieves semantically-based observed features. Third, an algorithm is developed to convert network bytes into images, and texture features are extracted by configuring an attention-based Residual Network (ResNet). Finally, the trained text and texture features are combined and used as multimodal features to classify various attacks. The proposed method is thoroughly evaluated on three widely used IoT-based datasets: CIC-IoT 2022, CIC-IoT 2023, and Edge-IIoT. The proposed method achieves excellent classification performance, with an accuracy of 98.2%. In addition, we present a game theory-based process to validate the proposed approach formally.

Review: Deep Learning and Fuzzy Logic Applications

The modeling and prediction field bosses a variety of practical applications, deep learning is a powerful tool used in this field. It has been proved that deep learning is a useful technique for extracting extremely accurate predictions from complex data sources, and also Recursive neural networks have demonstrated their usefulness in language translation and caption production, but convolutional neural networks remain the dominant solution for image classification tasks. In addition, deep learning, also known as deep neural networks, involves training models with multiple layers of interconnected artificial neurons. The primary idea of deep learning is to learn data representations through raising levels of abstraction. These strategies are effective, but they don't explain how the result is produced. Without knowing how a solution is arrived at using deep learning. In the field of artificial intelligence, deep learning and fuzzy logic are two powerful techniques. In addition, fuzzy logic combines deep learning will help deep learning select the desired features and work without supervision, this will make it possible to develop reliable systems with rich DL information even in the absence of hand-labeled data. Fuzzy logic that interpreted these features will subsequently provide explanations for the system's choice of classification label. This survey highlights the various applications which use fuzzy logic to improve deep learning.

Cross-layer self-representation enhanced deep subspace clustering with self-supervision

Deep subspace clustering typically employs a self-representation layer to accurately capture the similarity relationships among data points. While current techniques enhance self-representation matrices through optimized network structures and balanced regularization strategies, they often overlook comprehensive learning of global data characteristics. Moreover, scalability to large datasets is limited due to the proportional growth of the self-representation matrix with dataset size, presenting computational challenges. To address these issues, we introduce the Cross-Layer Self-Representation Enhanced Deep Subspace Clustering with Self-Supervision method. This novel approach features a hierarchical self-representation fusion mechanism that enriches the understanding of data relationships across different layers. In addition, we employ a contrastive learning strategy to refine data representation learning further. Crucially, we develop a joint framework that surmounts previous limitations in processing large-scale data. Our method's effectiveness and superiority are conclusively demonstrated through extensive experimental validation.

CDRM: Causal disentangled representation learning for missing data

Missing data pose significant challenges during representation learning of observational data. The incompleteness of data can result in a deterioration of generative performance in disentangled representation learning. Conventional data imputation solutions, such as regression imputation or multiple imputation, often neglect the underlying causal relationships among the data. To address these issues, the causal disentangled representation learning for missing data (CDRM) framework was proposed. Missing data with graph representations are integrated to construct heterogeneous networks composed of observations, features, and known feature values. To achieve data completeness, an interaction module consisting of a parallel neighbor interaction layer and an embedded update layer are integrated with the heterogeneous network to predict missing values. To recover the true causal relationship of missing data, edge embeddings are further introduced during message passing of heterogeneous networks, which can capture the interaction of different features and enrich the representation of observations. Furthermore, the causal relationship is incorporated into the VAE using two distinct encoders to learn representations of causally related concepts. In a series of experimental evaluations on diverse datasets, CDRM consistently outperforms the state-of-the-art method, namely, CausalVAE, in disentangled representation learning, particularly in scenarios with limited labeled data. Notably, CDRM can generate counterfactual data in response to missing data, further enhancing its utility in machine learning applications. The source code and data are available at https://github.com/Causal-Disentangled/CDRM.

Enhancing Time Series Predictability via Structure‐Aware Reservoir Computing

Accurate prediction of the future evolution of observational time series is a paramount challenge in current data‐driven research. While existing techniques struggle to learn useful representations from the temporal correlations, the high dimensionality in spatial domain is always considered as obstacle, leading to the curse of dimensionality and excessive resource consumption. This work designs a novel structure‐aware reservoir computing aiming at enhancing the predictability of coupled time series, by incorporating their historical dynamics as well as structural information. Paralleled reservoir computers with redesigned mixing inputs based on spatial relationships are implemented to cope with the multiple time series, whose core idea originates from the principle of the celebrated Granger causality. Representative numerical simulations and comparisons demonstrate the superior performance of the approach over the traditional ones. This work provides valuable insights into deeply mining both temporal and spatial information to enhance the representation learning of data in various machine learning techniques.

Towards Improved Privacy in Digital Marketing: A Unified Approach to User Modeling with Deep Learning on a Data Monetization Platform

This paper introduces an innovative method for safeguarding user privacy in digital marketing campaigns through the application of deep learning techniques on a data monetization platform. This framework empowers users to maintain authority over their personal data while enabling marketers to pinpoint suitable target audiences. The system consists of several key stages Data representation learning in hyperbolic space captures latent user interests across various data sources with hierarchical structures. Subsequently, Generative Adversarial Networks generate synthetic user interests from these embedding. To preserve user privacy, Federated Learning is utilized for decentralized user monetization, Data privacy, modeling training, ensuring data remains undisclosed to marketers. Lastly, a hyperbolic embedding, Federated learning targeting strategy, rooted in recommendation systems, utilizes learned user interests to identify optimal target audiences for digital marketing campaigns. In sum, this approach offers a comprehensive solution for privacy-preserving user modeling in digital marketing.

Latent Representation Learning for Geospatial Entities

Representation learning has been instrumental in the success of machine learning, offering compact and performant data representations for diverse downstream tasks. In the spatial domain, it has been pivotal in extracting latent patterns from various data types, including points, polylines, polygons, and networked structures. However, existing approaches often fall short of explicitly capturing both semantic and spatial information, relying on proxies and synthetic features. This paper presents GeoNN, a novel graph neural network-based model designed to learn spatially-aware embeddings for geospatial entities. GeoNN leverages edge features generated from geodesic functions, dynamically selecting relevant features based on relative locations. It introduces both transductive (GeoNN-T) and inductive (GeoNN-I) models, ensuring effective encoding of geospatial features and scalability with entity changes. Extensive experiments demonstrate GeoNN’s effectiveness in location-sensitive superpixel-based graphs and real-world points of interest, outperforming baselines across various evaluation measures.

Factor annealing decoupling compositional training method for imbalanced hyperspectral image classification

AbstractDue to differences in the quantity and size of observed targets, hyperspectral images are characterized by class imbalance. The standard deep learning classification model training scheme optimizes the overall classification error, which may lead to performance imbalance between classes in hyperspectral image classification frameworks. Therefore, a novel factor annealing decoupling compositional training method is proposed in this paper. Without requiring resampling or reweighting, it implicitly modulates the training process, so standard models can sufficiently learn the representation of the minority classes and further be trained as robust classifiers. Specifically, the label‐distribution‐aware margin loss is combined with the error‐rate‐based cross‐entropy loss via combination factor, which considers both imbalanced data representation learning and classifier overall performance. Then, a factor annealing optimization training scheme is designed to adjust the combination factor, which solves the stage division problem of two‐stage decoupling learning. Experimental results on two hyperspectral image datasets demonstrate that, as compared with other competing approaches, the proposed method can continuously and stably optimize the model parameters, achieving improvements in class average metrics and difficult classes without affecting overall classification performance.

Data Representation for Deep Learning - Based Arabic Text Summarization Performance Using Python Results

A sequence-to-sequence model is used as the foundation for a suggested abstractive Arabic text summarizing system. Our goal is to create a sequence-to-sequence model by utilizing multiple deep artificial neural networks and determining which one performs the best. The encoder and decoder have been developed using several layers of recurrent neural networks, gated recurrent units, recursive neural networks, convolutional neural networks, long short-term memory, and bidirectional long short-term memory. We are re-implementing the fundamental summarization model in this study, which uses the sequence-to-sequence framework. Using a Google Colab Jupiter notebook that runs smoothly, we have constructed these models using the Keras library. The results further demonstrate that one of the key techniques that has led to breakthrough performance with deep neural networks is the use of Gensim for word embeddings over other text representations by abstractive summarization models, along with FastText, a library for efficient learning of word representations and sentence classification.

Generation-based Multi-view Contrast for Self-supervised Graph Representation Learning

Graph contrastive learning has made remarkable achievements in the self-supervised representation learning of graph-structured data. By employing perturbation function (i.e., perturbation on the nodes or edges of graph), most graph contrastive learning methods construct contrastive samples on the original graph. However, the perturbation-based data augmentation methods randomly change the inherent information (e.g., attributes or structures) of the graph. Therefore, after nodes embedding on the perturbed graph, we cannot guarantee the validity of the contrastive samples as well as the learned performance of graph contrastive learning. To this end, in this article, we propose a novel generation-based multi-view contrastive learning framework (GMVC) for self-supervised graph representation learning, which generates the contrastive samples based on our generator rather than perturbation function. Specifically, after nodes embedding on the original graph we first employ random walk in the neighborhood to develop multiple relevant node sequences for each anchor node. We then utilize the transformer to generate the representations of relevant contrastive samples of anchor node based on the features and structures of the sampled node sequences. Finally, by maximizing the consistency between the anchor view and the generated views, we force the model to effectively encode graph information into nodes embeddings. We perform extensive experiments of node classification and link prediction tasks on eight benchmark datasets, which verify the effectiveness of our generation-based multi-view graph contrastive learning method.

Dual auto-weighted multi-view clustering via autoencoder-like nonnegative matrix factorization

Multi-view clustering (MVC) can exploit the complementary information among multi-view data to achieve the satisfactory performance, thus having extensive potentials for practical applications. Although Nonnegative Matrix Factorization (NMF) has emerged as an effective technique for MVC, the existing NMF-based methods still have two main limitations: 1) They solely focus on the reconstruction of original data, which can be regarded as the decoder of an autoencoder, while neglecting the low-dimensional representation learning. 2) They lack the ability to effectively capture both linear and nonlinear structures of data. To solve these problems, in this paper, we propose a Dual Auto-weighted multi-view clustering model based on Autoencoder-like NMF (DA2NMF), which enables a comprehensive exploration of both linear and nonlinear structures. Specifically, we establish an autoencoder-like NMF model that learns linear low-dimensional representations by integrating data reconstruction and representation learning within a unified framework. Moreover, the adaptive graph learning is introduced to explore the nonlinear structures in data. We further design a dual auto-weighted strategy to adaptively compute weights for different views and low-dimensional representations, thereby obtaining an enhanced consistent graph. An effective algorithm based on Multiplicative Update Rule (MUR) is developed to solve the DA2NMF with the theoretical convergence guarantee. Experimental results show that the proposed DA2NMF can effectively improve the clustering performance compared with the state-of-the-art MVC algorithms.

Pneumonia prediction on chest x-ray images using deep learning approach

<span lang="EN-US">Coronavirus disease 2019 (COVID-19) is an infectious disease with first symptoms similar to the flu. In many cases, this disease causes pneumonia. Since pulmonary infections can be observed through radiography images, this paper investigates deep learning methods for automatically analyzing query chest x-ray images. In deep learning, computers can automatically identify useful features for the model, directly from the raw data, bypassing the difficult step of manual information refinement. The main part of the deep learning method is the focus on automatically learning data representations. Visual geometry group 16 (VGG16) and DenseNet121 are methods in deep learning. The data used is a chest x-ray of pneumonia. Data is divided into training, testing, and validation. The best method for this research case is VGG16 with 93% accuracy training and 90% accuracy testing. In this study, DenseNet121 obtained accuracy below VGG16, with 92% accuracy in training and 88% for accuracy testing. Parameters have a significant influence on the accuracy of each model, and with the parameters that have been used, the VGG16 is a method that has high accuracy and can be used to predict chest x-ray images aimed at checking pneumonia in patients.</span>

Usage-aware representation learning for critical information identification in transportation networks

Extracting meaningful information from noisy high-dimensional data is attracting increasing attention as richer and higher resolution data is being collected and used for transportation system planning and management purposes. Discovering critical information via effective data representation learning not only helps reduce data dimension, it also enables a deeper understanding of the underlying properties of noisy data, which could then lead to better planning and operations decisions. In this study, we present a new perspective that, unlike most existing approaches in the general data science literature, the design of data representation should go beyond the data itself; it should incorporate an understanding of how the data is used in the domain-specific applications. We further argue that this design philosophy is particularly important for transportation data because of the high spatial correlations of transportation data brought by network interdependence. We propose a usage-aware representation learning framework by incorporating the information loss for downstream application into the data encoding-decoding process. The proposed approach is formulated as a Stiefel manifold optimization problem. The effectiveness of the proposed framework is demonstrated in two network applications: modeling transportation network flows and estimating network-level vehicular emissions. The performance of the learned representation from our approach is compared with existing approaches using multiple evaluation context, including data reconstruction quality, clustering, anomaly detection, and critical information identification, through case studies implemented in Sioux Falls, Boston, and San Jose networks. The good performance of our approach consistently observed in those experiments indicates the importance of incorporating the downstream data usage in the process of data representation learning.

Adaptively-Accelerated Parallel Stochastic Gradient Descent for High-Dimensional and Incomplete Data Representation Learning

Adaptively-Accelerated Parallel Stochastic Gradient Descent for High-Dimensional and Incomplete Data Representation Learning

Text clustering based on pre-trained models and autoencoders.

Text clustering is the task of grouping text data based on similarity, and it holds particular importance in the medical field. sIn healthcare, medical data clustering is a highly active and effective research area. It not only provides strong support for making correct medical decisions from medical datasets but also aids in patient record management and medical information retrieval. With the development of the healthcare industry, a large amount of medical data is being generated, and traditional medical data clustering faces significant challenges. Many existing text clustering algorithms are primarily based on the bag-of-words model, which has issues such as high dimensionality, sparsity, and the neglect of word positions and context. Pre-trained models are a deep learning-based approach that treats text as a sequence to accurately capture word positions and context information. Moreover, compared to traditional K-means and fuzzy C-means clustering models, deep learning-based clustering algorithms are better at handling high-dimensional, complex, and nonlinear data. In particular, clustering algorithms based on autoencoders can learn data representations and clustering information, effectively reducing noise interference and errors during the clustering process. This paper combines pre-trained language models with deep embedding clustering models. Experimental results demonstrate that our model performs exceptionally well on four public datasets, outperforming most existing text clustering algorithms, and can be applied to medical data clustering.

ScBiG for representation learning of single-cell gene expression data based on bipartite graph embedding.

Analyzing single-cell RNA sequencing (scRNA-seq) data remains a challenge due to its high dimensionality, sparsity and technical noise. Recognizing the benefits of dimensionality reduction in simplifying complexity and enhancing the signal-to-noise ratio, we introduce scBiG, a novel graph node embedding method designed for representation learning in scRNA-seq data. scBiG establishes a bipartite graph connecting cells and expressed genes, and then constructs a multilayer graph convolutional network to learn cell and gene embeddings. Through a series of extensive experiments, we demonstrate that scBiG surpasses commonly used dimensionality reduction techniques in various analytical tasks. Downstream tasks encompass unsupervised cell clustering, cell trajectory inference, gene expression reconstruction and gene co-expression analysis. Additionally, scBiG exhibits notable computational efficiency and scalability. In summary, scBiG offers a useful graph neural network framework for representation learning in scRNA-seq data, empowering a diverse array of downstream analyses.

Feature extraction and representation learning of 3D point cloud data

Three-dimensional point cloud data serves as a critical source of information in various real-world application domains, such as computer vision, robotics, geographic information systems, and medical image processing. Due to the discrete and unordered nature of point clouds, applying 2D image feature extractors directly to the extraction of 3D point cloud features is challenging. Therefore, we propose a novel variational feature component extraction method called PointFEA. This paper aims to research and propose a series of methods to enhance the feature extraction and representation learning of 3D point cloud data. Firstly, in terms of feature extraction, local neighborhood encoding is combined with the local latent representation of point clouds to obtain more correlated point cloud features. Secondly, in the domain of point cloud representation learning, the multi-scale representation learning method maps point cloud data into a high-dimensional space to better capture critical features and adapt to different granularities of point cloud data. Lastly, features of different dimensions are input into a cross-fusion transformer to obtain local attention coefficients. We validate our methods on commonly used point cloud datasets, and the experiments demonstrate the effectiveness of our approach, achieving accuracies of 94.8% on ModelNet40 and 89.1% on ScanObjectNN.

Hyperparameter Recommendation Integrated With Convolutional Neural Network.

Hyperparameter recommendation via meta-learning has shown great promise in various studies. The main challenge for meta-learning is how to develop an effective meta-learner (learning algorithm) that can capture the intrinsic relationship between dataset characteristics and the empirical performance of hyperparameters. Existing meta-learners are mostly based on traditional machine-learning models that only learn data representations with a single layer, which are incapable of learning complex features from the data and often cannot capture those properties deeply embedded in data. To address this issue, in this article, we propose hyperparameter recommendation approaches by integrating the learning model with convolutional neural networks (CNNs). Specifically, we first formulate the recommendation task as a regression problem, where dataset characteristics are treated as predictors and the historical performance of hyperparameters as responses. We establish a CNN-based learning model with feature selection capability to serve as the regressor. We then develop a convolutional denoising autoencoder (ConvDAE) that can leverage the spatial structure of the entire hyperparameter performance space and evaluate the performance of hyperparameters via denoising when the performance of partial hyperparameters is available under the multidimensional framework. To make our approach being flexible in applications, we establish a comprehensive two-branch CNN model that can utilize both dataset characteristics and partial evaluations to make effective recommendations. We conduct extensive experiments on 400 real classification problems and the well-known SVM. Our proposed approaches outperform existing meta-learning baselines as well as various search algorithms, demonstrating the high effectiveness in hyperparameter recommendations via deep learning.