Low-dimensional Representation Research Articles

An objective identification technique for potential vorticity structures associated with African easterly waves

Abstract. Tropical Africa and the North Atlantic Ocean are significantly influenced by African easterly waves (AEWs), which play a fundamental role in tropical rainfall and cyclogenesis in that region. The dynamics of AEWs can be described in a potential vorticity (PV) framework. The important impact of latent heat release by cloud processes is captured in this framework by the diabatic generation of PV anomalies. This paper introduces an innovative approach for the identification and tracking of PV structures within AEWs. By employing AEW tracking and computing the wave phase of each point within the AEW domain using a Hilbert transform, we are able to effectively identify and collect 3-D PV structures associated with specific AEWs. To facilitate a climatological analysis, performed here over the months of June to October from 2002 to 2022, these structures are subsequently characterized by low-dimensional descriptors, including their location, intensity, and orientation. Our climatological analysis reveals the seasonal evolution and the structural attributes of PV anomalies within AEWs over the study domain. PV feature locations closely align with the African easterly jet's latitudinal shift during the summer season. Analysis of the mean pressure level of the 3-D PV structures shows a remarkable shift during their life cycle, indicating deep moist convection characteristics over land and more shallow convection characteristics over the ocean. On average, PV features identified within AEW troughs tilt downshear over land and equatorward over the ocean. The trough-centered analysis reveals distinct differences between satellite-estimated and model-predicted rainfall. Agreement between the results of a more traditional composite analysis and our new feature analysis provides confidence in our feature approach as a novel diagnostic tool. The feature framework provides a low-dimensional representation of the PV structure of AEWs, which facilitates future statistical analyses of the relation of this structure to, for example, tropical cyclogenesis or the predictability of tropical rainfall.

Explaining deep learning-based representations of resting state functional connectivity data: focusing on interpreting nonlinear patterns in autism spectrum disorder

BackgroundResting state Functional Magnetic Resonance Imaging fMRI (rs-fMRI) has been used extensively to study brain function in psychiatric disorders, yielding insights into brain organization. However, the high dimensionality of the rs-fMRI data presents significant challenges for data analysis. Variational autoencoders (VAEs), a type of neural network, have been instrumental in extracting low-dimensional latent representations of resting state functional connectivity (rsFC) patterns, thereby addressing the complex nonlinear structure of rs-fMRI data. Despite these advances, interpreting these latent representations remains a challenge. This paper aims to address this gap by developing explainable VAE models and testing their utility using rs-fMRI data in autism spectrum disorder (ASD).MethodsOne-thousand one hundred and fifty participants (601 healthy controls [HC] and 549 patients with ASD) were included in the analysis. RsFC correlation matrices were extracted from the preprocessed rs-fMRI data using the Power atlas, which includes 264 regions of interest (ROIs). Then VAEs were trained in an unsupervised manner. Lastly, we introduce our latent contribution scores to explain the relationship between estimated representations and the original rs-fMRI brain measures.ResultsWe quantified the latent contribution scores for both the ASD and HC groups at the network level. We found that both ASD and HC groups share the top network connectivitives contributing to all estimated latent components. For example, latent 0 was driven by rsFC within ventral attention network (VAN) in both the ASD and HC. However, we found significant differences in the latent contribution scores between the ASD and HC groups within the VAN for latent 0 and the sensory/somatomotor network for latent 2.ConclusionThis study introduced latent contribution scores to interpret nonlinear patterns identified by VAEs. These scores effectively capture changes in each observed rsFC feature as the estimated latent representation changes, enabling an explainable deep learning model that better understands the underlying neural mechanisms of ASD.

A Fusion Learning Model Based on Deep Learning for Single-Cell RNA Sequencing Data Clustering.

Single-cell RNA sequencing (scRNA-seq) technology provides a means for studying biology from a cellular perspective. The fundamental goal of scRNA-seq data analysis is to discriminate single-cell types using unsupervised clustering. Few single-cell clustering algorithms have taken into account both deep and surface information, despite the recent slew of suggestions. Consequently, this article constructs a fusion learning framework based on deep learning, namely scGASI. For learning a clustering similarity matrix, scGASI integrates data affinity recovery and deep feature embedding in a unified scheme based on various top feature sets. Next, scGASI learns the low-dimensional latent representation underlying the data using a graph autoencoder to mine the hidden information residing in the data. To efficiently merge the surface information from raw area and the deeper potential information from underlying area, we then construct a fusion learning model based on self-expression. scGASI uses this fusion learning model to learn the similarity matrix of an individual feature set as well as the clustering similarity matrix of all feature sets. Lastly, gene marker identification, visualization, and clustering are accomplished using the clustering similarity matrix. Extensive verification on actual data sets demonstrates that scGASI outperforms many widely used clustering techniques in terms of clustering accuracy.

Designing meaningful continuous representations of T cell receptor sequences with deep generative models

T Cell Receptor (TCR) antigen binding underlies a key mechanism of the adaptive immune response yet the vast diversity of TCRs and the complexity of protein interactions limits our ability to build useful low dimensional representations of TCRs. To address the current limitations in TCR analysis we develop a capacity-controlled disentangling variational autoencoder trained using a dataset of approximately 100 million TCR sequences, that we name TCR-VALID. We design TCR-VALID such that the model representations are low-dimensional, continuous, disentangled, and sufficiently informative to provide high-quality TCR sequence de novo generation. We thoroughly quantify these properties of the representations, providing a framework for future protein representation learning in low dimensions. The continuity of TCR-VALID representations allows fast and accurate TCR clustering and is benchmarked against other state-of-the-art TCR clustering tools and pre-trained language models.

IMNE: Maximizing influence through deep learning-based node embedding in social network

Influence Maximization (IM) is a critical problem in social network analysis and marketing. It involves identifying a subset of nodes in a social network whose activation or influence can lead to the maximal spread of information, ideas, or behaviors within the network. Although many approaches have been developed in the literature to deal with this problem, most of these approaches are ineffective in dealing with large-scale social networks due to free parameters and computational complexity. Embeddings are used to learn low-dimensional representations of nodes in a social network. These embeddings capture the structural and semantic information of nodes and their relationships within the network. By training deep learning models on graph-structured data, node embeddings can capture complex patterns and dependencies in social networks, enabling more effective downstream tasks such as IM. Accordingly, this paper proposes an efficient algorithm to address the IM problem in social networks using deep learning-based Node Embedding (IMNE), which includes shell decomposition, graph/node embedding, and search space reduction as well as the use of local structural features. Our approach combines the power of deep learning for representation learning with the rich structural information present in social networks to address the challenge of IM in complex and dynamic social networks. IMNE uses the Independent Cascade (IC) information diffusion model to determine the labels needed to train the model by calculating the influence of nodes. Experimental results on several real-world networks considering different performance metrics show that IMNE performs better compared to existing baseline and state-of-the-art methods.

Application of Music Retrieval & Generation

The research explores music generation through LSTM and VAEs neural network architectures, leveraging MIDI representations. LSTM specializes in sequential data processing, while VAEs compress music datasets into low-dimensional representations. Optimization of models is pursued by analyzing the relationship between training loss and epochs. Ultimately, a comparison between LSTM and VAEs determines the most effective algorithm for music generation. Traditional methods face issues like vanishing and exploding gradients, prompting exploration of deep learning approaches. The study aims to advance machine manipulation of music, facilitating new composition generation from existing MIDI files. Keywords: Music generation, LSTM, Variational Autoencoders (VAEs), MIDI representation, Sequential data, Neural network architectures, Training loss, Optimization, Deep learning, Vanishing gradient, Exploding gradient

Combining EEG Features and Convolutional Autoencoder for Neonatal Seizure Detection.

Neonatal epilepsy is a common emergency phenomenon in neonatal intensive care units (NICUs), which requires timely attention, early identification, and treatment. Traditional detection methods mostly use supervised learning with enormous labeled data. Hence, this study offers a semi-supervised hybrid architecture for detecting seizures, which combines the extracted electroencephalogram (EEG) feature dataset and convolutional autoencoder, called Fd-CAE. First, various features in the time domain and entropy domain are extracted to characterize the EEG signal, which helps distinguish epileptic seizures subsequently. Then, the unlabeled EEG features are fed into the convolutional autoencoder (CAE) for training, which effectively represents EEG features by optimizing the loss between the input and output features. This unsupervised feature learning process can better combine and optimize EEG features from unlabeled data. After that, the pre-trained encoder part of the model is used for further feature learning of labeled data to obtain its low-dimensional feature representation and achieve classification. This model is performed on the neonatal EEG dataset collected at the University of Helsinki Hospital, which has a high discriminative ability to detect seizures, with an accuracy of 92.34%, precision of 93.61%, recall rate of 98.74%, and F1-score of 95.77%, respectively. The results show that unsupervised learning by CAE is beneficial to the characterization of EEG signals, and the proposed Fd-CAE method significantly improves classification performance.

Efficient identification of broad absorption line quasars using dimensionality reduction and machine learning

Efficient identification of broad absorption line quasars using dimensionality reduction and machine learning

Cascaded maximum median-margin discriminant projection with its application to face recognition

This paper introduces a novel method called cascaded maximum median-margin discriminant projection (CMMDP) for robust face recognition. Initially, we introduce the maximum median-margin discriminant projection (MMDP) method, which aims to maximize the distances between class medians and their inter-class neighbors, while simultaneously minimizing the distances between class medians and their intra-class samples. To extract features at multiple levels, we extend the MMDP method to its deep version, known as CMMDP. The CMMDP algorithm incorporates the MMDP technique to learn multistage filter banks specifically designed for facial images. Afterward, we utilize straightforward binary hashing techniques for efficient indexing, along with block histograms for effective feature pooling. The CMMDP model is simple and easy to train, while taking full account of the global and local structure of the samples by maximizing the distances between samples that are dissimilar and minimizing the distances between samples that are similar after projection. This results in a low-dimensional data representation with better discriminative performance. We evaluate the performance of CMMDP on face datasets including AR, ORL, ExtYaleB and CMU PIE databases. The results indicate that CMMDP achieves superior performance than most advanced methods.

Comparison of different group-level templates in gradient-based multimodal connectivity analysis

Abstract The study of large-scale brain connectivity is increasingly adopting unsupervised approaches that derive low-dimensional spatial representations from high-dimensional connectomes, referred to as gradient analysis. When translating this approach to study interindividual variations in connectivity, one technical issue pertains to the selection of an appropriate group-level template to which individual gradients are aligned. Here, we compared different group-level template construction strategies using functional and structural connectome data from neurotypical controls and individuals with autism spectrum disorder (ASD) to identify between-group differences. We studied multimodal magnetic resonance imaging data obtained from the Autism Brain Imaging Data Exchange (ABIDE) Initiative II and the Human Connectome Project (HCP). We designed six template construction strategies that varied in whether: (1) they included typical controls in addition to ASD; or (2) they mapped from one dataset onto another. We found that aligning a combined subject template of the ASD and control subjects from the ABIDE Initiative onto the HCP template exhibited the most pronounced effect size. This strategy showed robust identification of ASD-related brain regions for both functional and structural gradients across different study settings. Replicating the findings on focal epilepsy demonstrated the generalizability of our approach. Our findings will contribute to improving gradient-based connectivity research.

Transferability of Machine Learning Models for Geogenic Contaminated Groundwaters.

Machine learning models show promise in identifying geogenic contaminated groundwaters. Modeling in regions with no or limited samples is challenging due to the need for large training sets. One potential solution is transferring existing models to such regions. This study explores the transferability of high fluoride groundwater models between basins in the Shanxi Rift System, considering six factors, including modeling methods, predictor types, data size, sample/predictor ratio (SPR), predictor range, and data informing. Results show that transferability is achieved only when model predictors are based on hydrochemical parameters rather than surface parameters. Data informing, i.e., adding samples from challenging regions to the training set, further enhances the transferability. Stepwise regression shows that hydrochemical predictors and data informing significantly improve transferability, while data size, SPR, and predictor range have no significant effects. Additionally, despite their stronger nonlinear capabilities, random forests and artificial neural networks do not necessarily surpass logistic regression in transferability. Lastly, we utilize the t-SNE algorithm to generate low-dimensional representations of data from different basins and compare these representations to elucidate the critical role of predictor types in transferability.

A domain knowledge-informed design space exploration methodology for mechanical layout design

Layout designs have a large number of design variables and various physical constraints. Conventional design exploration approaches are time-consuming and may require human intervention. A unique feature about physical layout designs is the availability of domain knowledge, which can be utilised to speed up the design process. In this paper, we propose a generative deep learning-based design space exploration (DSE) methodology that is capable of learning design constraints in the layout design without explicit supervision. Moreover, it can incorporate domain knowledge in the generated layouts, thereby speeding up the design process. This is realised by constructing a layout generation variational autoencoder (LGVAE) model, which uses a latent space as an interface to generate the layouts. By training the LGVAE model, significantly lower-dimensional representations can be learned compared to the original dimensionality of the design space. Therefore, the number of design variables is greatly reduced. We showcase the performance of the proposed DSE approach by solving the heat source layout design problem encountered in thermal management of chips. Experiments demonstrate that the LGVAE model is capable of generating compressed latent representations that capture the characteristics of the input samples, which makes the DSE cost-effective.

CarePlus: A general framework for hardware performance counter based malware detection under system resource competition

Hardware performance counter (HPC) has been used for malware detection because of its lightweight overhead. Unfortunately, such detection models including their software implementation and hardware implementation suffer from performance decline in environments where resource competition among programs exists in the operating system. In this paper, we propose a framework called CarePlus, which enables two kinds of implementations of hardware performance counter-based malware detection (HMD) resilient to resource competition. The core idea is to detect malware by using invariants extracted from HPC-level behaviors in resource competition environments. To achieve this, a benchmark-based resource pressure generator is designed to generate controlled resource competition environments for observing typical HPC-level behavior of a program. Then, a behavior representation network is trained to map HPC-level behaviors in any competition environment into low-dimensional representations, which allows software-implemented HMD models built on them to detect malware regardless of resource competition. Finally, an adapter is introduced to project behavior representation in the competition environment into data that conforms to the training distribution of the hardware-implemented HMD model. Using three datasets collected in different application systems (e.g., server or desktop) with different resource competition types or levels, the experimental results show that compared with existing detection models, CarePlus is helpful to improve the performance of software-implemented HMD classifier and hardware-implemented HMD classifier in the competition environment. We also prove that the classifier using CarePlus is insensitive to different application systems. Finally, we demonstrate the computational overhead.

ClusterLP: A novel Cluster-aware Link Prediction model in undirected and directed graphs

Link prediction models endeavor to understand the distribution of links within graphs and forecast the presence of potential links. With the advancements in deep learning, prevailing methods typically strive to acquire low-dimensional representations of nodes in networks, aiming to capture and retain the structure and inherent characteristics of networks. However, the majority of these methods primarily focus on preserving the microscopic structure, such as the first- and second-order proximities of nodes, while largely disregarding the mesoscopic cluster structure, which stands out as one of the network's most prominent features. Following the homophily principle, nodes within the same cluster exhibit greater similarity to each other compared to those from different clusters, suggesting that they should possess analogous vertex representations and higher probabilities of linkage. In this study, we develop a straightforward yet efficient Cluster-aware Link Prediction framework (ClusterLP), with the objective of directly leveraging cluster structures to predict links among nodes with maximum accuracy in both undirected and directed graphs. Specifically, we posit that establishing links between nodes with similar representation vectors and cluster tendencies is more feasible in undirected graphs, whereas nodes in directed graphs are inclined to point towards nodes with akin representation vectors and greater influence. We tailor the implementation of ClusterLP for undirected and directed graphs, respectively, and experimental findings using multiple real-world networks demonstrate the high competitiveness of our models in the realm of link prediction tasks. The code utilized in our implementation is accessible at https://github.com/ZINUX1998/ClusterLP.

NCLDR: Nearest-Neighbor Contrastive Learning with Dual Correlation Loss for Dimensionality Reduction

Dimensionality reduction is an efficient method for alleviating the issue of dimensionality in high-dimensional data. As a popular self-supervised learning method, contrastive learning has recently garnered considerable attention. In this paper, we propose NCLDR: Nearest-Neighbor Contrastive Learning with Dual Correlation Loss for Dimensionality Reduction, a novel dimensionality reduction method that is porting a contrastive learning framework to the specific task of dimensionality reduction. Firstly, NCLDR uses the nearest-neighbor to construct feature pairs from the training set itself. Afterwards, to decorrelate feature variables that produce representations invariant across such pairs, a basic multi-layer perceptron (MLP) network architecture with a dual correlation loss function is designed. Compared to most dimensionality reduction methods, NCLDR bypasses the complexity of optimizing kNN graphs and facilitates the embedding of out-of-sample data. Additionally, it also alleviates the issue of “dimensional collapse” in the low-dimensional representation space. Finally, experimental results demonstrate that the proposed method achieves significant improvements over state-of-the-art dimensionality reduction methods.

The effect of data transformation on low-dimensional integration of single-cell RNA-seq

BackgroundRecent developments in single-cell RNA sequencing have opened up a multitude of possibilities to study tissues at the level of cellular populations. However, the heterogeneity in single-cell sequencing data necessitates appropriate procedures to adjust for technological limitations and various sources of noise when integrating datasets from different studies. While many analysis procedures employ various preprocessing steps, they often overlook the importance of selecting and optimizing the employed data transformation methods.ResultsThis work investigates data transformation approaches used in single-cell clustering analysis tools and their effects on batch integration analysis. In particular, we compare 16 transformations and their impact on the low-dimensional representations, aiming to reduce the batch effect and integrate multiple single-cell sequencing data. Our results show that data transformations strongly influence the results of single-cell clustering on low-dimensional data space, such as those generated by UMAP or PCA. Moreover, these changes in low-dimensional space significantly affect trajectory analysis using multiple datasets, as well. However, the performance of the data transformations greatly varies across datasets, and the optimal method was different for each dataset. Additionally, we explored how data transformation impacts the analysis of deep feature encodings using deep neural network-based models, including autoencoder-based models and proto-typical networks. Data transformation also strongly affects the outcome of deep neural network models.ConclusionsOur findings suggest that the batch effect and noise in integrative analysis are highly influenced by data transformation. Low-dimensional features can integrate different batches well when proper data transformation is applied. Furthermore, we found that the batch mixing score on low-dimensional space can guide the selection of the optimal data transformation. In conclusion, data preprocessing is one of the most crucial analysis steps and needs to be cautiously considered in the integrative analysis of multiple scRNA-seq datasets.

Particle Swarm Optimization for 5g and Beyond

Network embedding assigns nodes in a network to low dimensional representations and effectively preserves the network structure. Recently, a significant amount of progresses has been made toward this emerging network analysis paradigm. In this survey, we focus on categorizing and then reviewing the current development on network embedding methods, and point out its future research directions. We first summarize the motivation of network embedding. Particle Swarm Optimization (PSO) algorithm is used as the proposed method. We discuss the classical graph embedding algorithms on cognitive radio environment and their relationship with network embedding. Afterwards and primarily, we provide a comprehensive overview of a large number of networks embedding methods in a systematic manner, covering the structure- and property-preserving network embedding methods, the network embedding methods with side information and the advanced information preserving network embedding methods. Moreover, several evaluation approaches for network embedding and some useful online resources, including the network data sets and software, are reviewed, too. Finally, we discuss the framework of exploiting these network embedding methods to build an effective system and point out some potential future directions. Key Word: Particle Swarm Optimization (PSO), Beyond 5G (B5G), Internet of Think (IOT), Machine Learning (ML)etc..

Representation learning of in-degree-based digraph with rich information

AbstractNetwork representation learning aims to map the relationship between network nodes and context nodes to a low-dimensional representation vector space. Directed network representation learning considers mapping directional of node vector. Currently, only sporadic work on direct network representation has been reported. In this work, we propose a novel algorithm that takes into account the direction of the edge with text’s attribute of the node in directed network representation learning. We then define the matrix based on in-degree of Laplacian and signless Laplacian for digraph, and it utilizes web page datasets from universities in the USA to evaluate the performance of vertex classification. We compare our algorithm with other directed representation learning algorithms. The experimental results show that our algorithm outperforms the baseline by over 20% when the training ratio ranges from 10 to 90%. We apply the in-degree-Laplacian and In-degree-signless-Laplacian to directed representation learning, which is one of the main contributions of this algorithm. Additionally, we incorporate text information through matrix completion in directed network representation learning and the experimental results show an increase in performance of up to 20% compared to the baseline, especially when the training ratio is 10%.

A deep learning-based pipeline for developing multi-rib shape generative model with populational percentiles or anthropometrics as predictors

Rib cross-sectional shapes (characterized by the outer contour and cortical bone thickness) affect the rib mechanical response under impact loading, thereby influence the rib injury pattern and risk. A statistical description of the rib shapes or their correlations to anthropometrics is a prerequisite to the development of numerical human body models representing target demographics. Variational autoencoders (VAE) as anatomical shape generators remain to be explored in terms of utilizing the latent vectors to control or interpret the representativeness of the generated results. In this paper, we propose a pipeline for developing a multi-rib cross-sectional shape generative model from CT images, which consists of the achievement of rib cross-sectional shape data from CT images using an anatomical indexing system and regular grids, and a unified framework to fit shape distributions and associate shapes to anthropometrics for different rib categories. Specifically, we collected CT images including 3193 ribs, surface regular grid is generated for each rib based on anatomical coordinates, the rib cross-sectional shapes are characterized by nodal coordinates and cortical bone thickness. The tensor structure of shape data based on regular grids enable the implementation of CNNs in the conditional variational autoencoder (CVAE). The CVAE is trained against an auxiliary classifier to decouple the low-dimensional representations of the inter- and intra- variations and fit each intra-variation by a Gaussian distribution simultaneously. Random tree regressors are further leveraged to associate each continuous intra-class space with the corresponding anthropometrics of the subjects, i.e., age, height and weight. As a result, with the rib class labels and the latent vectors sampled from Gaussian distributions or predicted from anthropometrics as the inputs, the decoder can generate valid rib cross-sectional shapes of given class labels (male/female, 2nd to 11th ribs) for arbitrary populational percentiles or specific age, height and weight, which paves the road for future biomedical and biomechanical studies considering the diversity of rib shapes across the population.

Graph Representations for Reinforcement Learning

Graph analysis is becoming increasingly important due to the expressive power of graph models and the efficient algorithms available for processing them. Reinforcement Learning is one domain that could benefit from advancements in graph analysis, given that a learning agent may be integrated into an environment that can be represented as a graph. Nevertheless, the structural irregularity of graphs and the lack of prior labels make it difficult to integrate such a model into modern Reinforcement Learning frameworks that rely on artificial neural networks. Graph embedding enables the learning of low-dimensional vector representations that are more suited for machine learning algorithms, while retaining essential graph features. This paper presents a framework for evaluating graph embedding algorithms and their ability to preserve the structure and relevant features of graphs by means of an internal validation metric, without resorting to subsequent tasks that require labels for training. Based on this framework, three defined algorithms that meet the necessary requirements for solving a specific problem of Reinforcement Learning in graphs are selected, analyzed, and compared. These algorithms are Graph2Vec, GL2Vec, and Wavelet Characteristics, with the latter two demonstrating superior performance.