Representation Learning Algorithm Research Articles

A Negative Sample-Free Graph Contrastive Learning Algorithm

Self-supervised learning is a new machine learning method that does not rely on manually labeled data, and learns from rich unlabeled data itself by designing agent tasks using the input data as supervision to obtain a more generalized representation for application in downstream tasks. However, the current self-supervised learning suffers from the problem of relying on the selection and number of negative samples and the problem of sample bias phenomenon after graph data augmentation. In this paper, we investigate the above problems and propose a corresponding solution, proposing a graph contrastive learning algorithm without negative samples. The model uses matrix sketching in the implicit space for feature augmentation to reduce sample bias and iteratively trains the mutual correlation matrix of two viewpoints by drawing closer to the distance of the constant matrix as the objective function. This method does not require techniques such as negative samples, gradient stopping, and momentum updating to prevent self-supervised model collapse. This method is compared with 10 graph representation learning algorithms on four datasets for node classification tasks, and the experimental results show that the algorithm proposed in this paper achieves good results.

Multimodal Moore-Penrose Inverse-Based Recomputation Framework for Big Data Analysis.

Most multilayer Moore-Penrose inverse (MPI)-based neural networks, such as deep random vector functional link (RVFL), are structured with two separate stages: unsupervised feature encoding and supervised pattern classification. Once the unsupervised learning is finished, the latent encoding is fixed without supervised fine-tuning. However, in complex tasks such as handling the ImageNet dataset, there are often many more clues that can be directly encoded, while unsupervised learning, by definition, cannot know exactly what is useful for a certain task. There is a need to retrain the latent space representations in the supervised pattern classification stage to learn some clues that unsupervised learning has not yet been learned. In particular, the residual error in the output layer is pulled back to each hidden layer, and the parameters of the hidden layers are recalculated with MPI for more robust representations. In this article, a recomputation-based multilayer network using Moore-Penrose inverse (RML-MP) is developed. A sparse RML-MP (SRML-MP) model to boost the performance of RML-MP is then proposed. The experimental results with varying training samples (from 3k to 1.8 million) show that the proposed models provide higher Top-1 testing accuracy than most representation learning algorithms. For reproducibility, the source codes are available at https://github.com/W1AE/Retraining.

Representation learning of in-degree-based digraph with rich information

Network representation learning aims to map the relationship between network nodes and context nodes to a low-dimensional representation vector space. Directed network representation learning considers mapping directional of node vector. Currently, only sporadic work on direct network representation has been reported. In this work, we propose a novel algorithm that takes into account the direction of the edge with text’s attribute of the node in directed network representation learning. We then define the matrix based on in-degree of Laplacian and signless Laplacian for digraph, and it utilizes web page datasets from universities in the USA to evaluate the performance of vertex classification. We compare our algorithm with other directed representation learning algorithms. The experimental results show that our algorithm outperforms the baseline by over 20% when the training ratio ranges from 10 to 90%. We apply the in-degree-Laplacian and In-degree-signless-Laplacian to directed representation learning, which is one of the main contributions of this algorithm. Additionally, we incorporate text information through matrix completion in directed network representation learning and the experimental results show an increase in performance of up to 20% compared to the baseline, especially when the training ratio is 10%.

Research on the design of teaching aid resource system based on unsupervised representation learning

In the absence of manual annotation information, using only the topology information of the graph to achieve unsupervised graph alignment has always been one of the important challenges faced by graph data mining, especially in the large-scale graph alignment task, the discovery of initial seed nodes and the low computational efficiency have always been a problem. This paper proposes a large-scale unsupervised graph-aligned physical education teaching resource system framework based on topological representation learning, which is highly scalable and adaptable. Firstly, representative subgraphs are selected from the graph to be matched as seed node candidate sets, and the local topology information is used to calculate to obtain highly reliable seed node matching results; then, the obtained seed nodes are used to integrate the graph to be matched, and an efficient unsupervised representation learning algorithm is proposed to map the fusion graph to a unified vector space; finally the obtained node vectors are used to achieve the alignment of large-scale graphs. The proposed technique shows the shortest time when dealing with large-scale graph alignment tasks, and at the same time reaches the highest level of accuracy of alignment results, in addition, the performance of the physical education auxiliary resource system algorithm has the least impact on the difference of graph structure.

RPS: Portfolio asset selection using graph based representation learning

Portfolio optimization is one of the essential fields of focus in finance. There has been an increasing demand for novel computational methods in this area to compute portfolios with better returns and lower risks in recent years. We present a novel computational method called Representation Portfolio Selection by redefining the distance matrix of financial assets using Representation Learning and Clustering algorithms for portfolio selection to increase diversification. RPS proposes a heuristic for getting closer to the optimal subset of assets. Using empirical results in this paper, we demonstrate that widely used portfolio optimization algorithms, such as Mean-Variance Optimization, Critical Line Algorithm, and Hierarchical Risk Parity can benefit from our asset subset selection.

Representation learning algorithms for inferring machine independent latent features in pedestals in JET and AUG

Variational autoencoder (VAE)-based representation learning algorithms are explored for their capability to disentangle tokamak size dependence from other dependencies in a dataset of thousands of observed pedestal electron density and temperature profiles from JET and ASDEX Upgrade tokamaks. Representation learning aims to establish a useful representation that characterizes the dataset. In the context of magnetic confinement fusion devices, a useful representation could be considered to map the high-dimensional observations to a manifold that represents the actual degrees of freedom of the plasma scenario. A desired property for these representations is organization of the information into disentangled variables, enabling interpretation of the latent variables as representations of semantically meaningful characteristics of the data. The representation learning algorithms in this work are based on VAE that encodes the pedestal profile information into a reduced dimensionality latent space and learns to reconstruct the full profile information given the latent representation. Attaching an auxiliary regression objective for the machine control parameter configuration, broadly following the architecture of the domain invariant variational autoencoder (DIVA), the model learns to associate device control parameters with the latent representation. With this multimachine dataset, the representation does encode density scaling with device size that is qualitatively consistent with Greenwald density limit scaling. However, if the major radius of the device is given through a common regression objective with the other machine control parameters, the latent state of the representation struggles to clearly disentangle the device size from changes of the other machine control parameters. When separating the device size as an independent latent variable with dedicated regression objectives, similar to separation of domain and class labels in the original DIVA publication, the latent space becomes well organized as a function of the device size.

On learning latent dynamics of the AUG plasma state

In this work, we demonstrate the utility of state representation learning applied to modeling the time evolution of electron density and temperature profiles at ASDEX-Upgrade (AUG). The proposed model is a deep neural network, which learns to map the high dimensional profile observations to a lower dimensional state. The mapped states, alongside the original profile's corresponding machine parameters, are used to learn a forward model to propagate the state in time. We show that this approach is able to predict AUG discharges using only a selected set of machine parameters. The state is then further conditioned to encode information about the confinement regime, which yields a simple baseline linear classifier, while still retaining the information needed to predict the evolution of profiles. We, then, discuss the potential use cases and limitations of state representation learning algorithms applied to fusion devices.

Multi-channel high-order network representation learning research.

The existing network representation learning algorithms mainly model the relationship between network nodes based on the structural features of the network, or use text features, hierarchical features and other external attributes to realize the network joint representation learning. Capturing global features of the network allows the obtained node vectors to retain more comprehensive feature information during training, thereby enhancing the quality of embeddings. In order to preserve the global structural features of the network in the training results, we employed a multi-channel learning approach to perform high-order feature modeling on the network. We proposed a novel algorithm for multi-channel high-order network representation learning, referred to as the Multi-Channel High-Order Network Representation (MHNR) algorithm. This algorithm initially constructs high-order network features from the original network structure, thereby transforming the single-channel network representation learning process into a multi-channel high-order network representation learning process. Then, for each single-channel network representation learning process, the novel graph assimilation mechanism is introduced in the algorithm, so as to realize the high-order network structure modeling mechanism in the single-channel network representation learning. Finally, the algorithm integrates the multi-channel and single-channel mechanism of high-order network structure joint modeling, realizing the efficient use of network structure features and sufficient modeling. Experimental results show that the node classification performance of the proposed MHNR algorithm reaches a good order on Citeseer, Cora, and DBLP data, and its node classification performance is better than that of the comparison algorithm used in this paper. In addition, when the vector length is optimized, the average classification accuracy of nodes of the proposed algorithm is up to 12.24% higher than that of the DeepWalk algorithm. Therefore, the node classification performance of the proposed algorithm can reach the current optimal order only based on the structural features of the network under the condition of no external feature supplementary modeling.

T-Distributed Stochastic Neighbor Embedding for Co-Representation Learning

Co-clustering is the simultaneous clustering of the samples and attributes of a data matrix that provides deeper insight into data than traditional clustering. However, there is a lack of representation learning algorithms that serve this mechanism of co-clustering, and the current representation learning algorithms are limited to the sample perspective and lack the use of information in the attribute perspective. To solve this problem, in this article, ctSNE , a co-representation learning model based on t-distributed stochastic neighbor embedding, is proposed for unsupervised co-clustering, where ctSNE makes the dataset representation outputted more discriminative of row and column clusters (i.e. co-discrimination). On the basis of t-distributed stochastic neighbor embedding retaining the sample data distribution and local data structure, the philosophy of collaboration is introduced (i.e., row and column hidden relationship information) so that the ctSNE model is equipped with co-representation learning capability, which can effectively improve the performance of co-clustering. To prove the effectiveness of the ctSNE model, several classic co-clustering algorithms are used to check the co-representation performance of ctSNE, and a novel internal index based on an internal clustering index, known as total inertia, is proposed to demonstrate the effect of co-clustering. The numerous experimental results show that ctSNE has tremendous co-representation capability and can significantly improve the performance of co-clustering algorithms.

A Market Convergence Prediction Framework Based on a Supply Chain Knowledge Graph

Market convergence challenges socially sustainable supply chain management (SSSCM) due to the increasing competition. Identifying market convergence trends allows companies to respond quickly to market changes and improve supply chain resilience (SCR). Conventional approaches are one-sided and biased and cannot predict market convergence trends comprehensively and accurately. To address this issue, we propose a framework based on info2vec that solves the problem of matching multidimensional data by using the technology layer as the focal layer and the supply chain as the supporting layer. The framework enriches the supply chain dimension with the technology dimension. A knowledge graph is constructed to facilitate cross-domain information connectivity by integrating different data sources. The nodes in the knowledge graph were characterized using a representation learning algorithm, which enhanced feature mining during supply chain and market convergence. Changes in market demand were predicted based on link prediction experiments. Market convergence has an impact on firm cooperation and, thus, on SCR. The framework recommends potential technological and innovative cooperation opportunities for firms. In this way, it has been demonstrated to improve SSSCM through network resilience experiments. This method predicts market convergence efficiently based on the supply chain knowledge graph, which provides decision support for enterprise development.

Enhanced Structure-Based Prediction of Chiral Stationary Phases for Chromatographic Enantioseparation from 3D Molecular Conformations.

The accurate prediction of suitable chiral stationary phases (CSPs) for resolving the enantiomers of a given compound poses a significant challenge in chiral chromatography. Previous attempts at developing machine learning models for structure-based CSP prediction have primarily relied on 1D SMILES strings [the simplified molecular-input line-entry system (SMILES) is a specification in the form of a line notation for describing the structure of chemical species using short ASCII strings] or 2D graphical representations of molecular structures and have met with only limited success. In this study, we apply the recently developed 3D molecular conformation representation learning algorithm, which uses rapid conformational analysis and point clouds of atom positions in the 3D space, enabling efficient chemical structure-based machine learning. By harnessing the power of the rapid 3D molecular representation learning and a data set comprising over 300,000 chromatographic enantioseparation records sourced from the literature, our models afford notable improvements for the chemical structure-based choice of appropriate CSP for enantioseparation, paving the way for more efficient and informed decision-making in the field of chiral chromatography.

Autoencoder in Autoencoder Networks.

Modeling complex correlations on multiview data is still challenging, especially for high-dimensional features with possible noise. To address this issue, we propose a novel unsupervised multiview representation learning (UMRL) algorithm, termed autoencoder in autoencoder networks (AE2-Nets). The proposed framework effectively encodes information from high-dimensional heterogeneous data into a compact and informative representation with the proposed bidirectional encoding strategy. Specifically, the proposed AE2-Nets conduct encoding in two directions: the inner-AE-networks extract view-specific intrinsic information (forward encoding), while the outer-AE-networks integrate this view-specific intrinsic information from different views into a latent representation (backward encoding). For the nested architecture, we further provide a probabilistic explanation and extension from hierarchical variational autoencoder. The forward-backward strategy flexibly addresses high-dimensional (noisy) features within each view and encodes complementarity across multiple views in a unified framework. Extensive results on benchmark datasets validate the advantages compared to the state-of-the-art algorithms.

Self-guided disentangled representation learning for single image dehazing

Image dehazing has received extensive research attention as images collected in hazy weather are limited by low visibility and information dropout. Recently, disentangled representation learning has made excellent progress in various vision tasks. However, existing networks for low-level vision tasks lack efficient feature interaction and delivery mechanisms in the disentanglement process or an evaluation mechanism for the degree of decoupling in the reconstruction process, rendering direct application to image dehazing challenging. We propose a self-guided disentangled representation learning (SGDRL) algorithm with a self-guided disentangled network to realize multi-level progressive feature decoupling through sharing and interaction. The self-guided disentangled (SGD) network extracts image features using the multi-layer backbone network, and attribute features are weighted using the self-guided attention mechanism for the backbone features. In addition, we introduce a disentanglement-guided (DG) module to evaluate the degree of feature decomposition and guide the feature fusion process in the reconstruction stage. Accordingly, we develop SGDRL-based unsupervised and semi-supervised single image dehazing networks. Extensive experiments demonstrate the superiority of the proposed method for real-world image dehazing. The source code is available at https://github.com/dehazing/SGDRL.

Improved graph representation learning based on neighborhood aggregation and interaction fusion

The adaptive fusion module with an attention mechanism functions by employing a dual-channel graph convolutional network to aggregate neighborhood information. The resulting embeddings are then utilized to calculate interaction terms, thereby incorporating additional information. To enhance the relevance of fusion information, an adaptive fusion module with an attention mechanism is constructed. This module selectively combines the neighborhood aggregation and interaction terms, prioritizing the most pertinent information. Through this adaptive fusion process, the algorithm effectively captures both neighborhood features and other nonlinear information, leading to improved overall performance. Neighborhood Aggregation Interaction Graph Convolutional Network Adaptive Fusion (NAIGCNAF) is a graph representation learning algorithm designed to obtain low-dimensional node representations while preserving graph properties. It addresses the limitations of existing algorithms, which tend to focus solely on aggregating neighborhood features and overlook other nonlinear information. NAIGCNAF utilizes a dual-channel graph convolutional network for neighborhood aggregation and calculates interaction terms based on the resulting embeddings. Additionally, it incorporates an adaptive fusion module with an attention mechanism to enhance the relevance of fusion information. Extensive evaluations on three citation datasets demonstrate that NAIGCNAF outperforms other algorithms such as GCN, Neighborhood Aggregation, and AIR-GCN. NAIGCNAF achieves notable improvements in classification accuracy, ranging from 1.0 to 1.6 percentage points on the Cora dataset, 1.1 to 2.4 percentage points on the Citeseer dataset, and 0.3 to 0.9 percentage points on the Pubmed dataset. Moreover, in visualization tasks, NAIGCNAF exhibits clearer boundaries and stronger aggregation within clusters, enhancing its effectiveness. Additionally, the algorithm showcases faster convergence rates and smoother accuracy curves, further emphasizing its ability to improve benchmark algorithm performance.

GNN Representation Learning and Multi-Objective Variable Neighborhood Search Algorithm for Wind Farm Layout Optimization

GNN Representation Learning and Multi-Objective Variable Neighborhood Search Algorithm for Wind Farm Layout Optimization

Toward Graph Self-Supervised Learning With Contrastive Adjusted Zooming.

Graph representation learning (GRL) is critical for graph-structured data analysis. However, most of the existing graph neural networks (GNNs) heavily rely on labeling information, which is normally expensive to obtain in the real world. Although some existing works aim to effectively learn graph representations in an unsupervised manner, they suffer from certain limitations, such as the heavy reliance on monotone contrastiveness and limited scalability. To overcome the aforementioned problems, in light of the recent advancements in graph contrastive learning, we introduce a novel self-supervised GRL algorithm via graph contrastive adjusted zooming, namely, G-Zoom, to learn node representations by leveraging the proposed adjusted zooming scheme. Specifically, this mechanism enables G-Zoom to explore and extract self-supervision signals from a graph from multiple scales: micro (i.e., node level), meso (i.e., neighborhood level), and macro (i.e., subgraph level). First, we generate two augmented views of the input graph via two different graph augmentations. Then, we establish three different contrastiveness on the above three scales progressively, from node, neighboring, to subgraph level, where we maximize the agreement between graph representations across scales. While we can extract valuable clues from a given graph on the micro and macro perspectives, the neighboring-level contrastiveness offers G-Zoom the capability of a customizable option based on our adjusted zooming scheme to manually choose an optimal viewpoint that lies between the micro and macro perspectives to better understand the graph data. In addition, to make our model scalable to large graphs, we use a parallel graph diffusion approach to decouple model training from the graph size. We have conducted extensive experiments on real-world datasets, and the results demonstrate that our proposed model outperforms the state-of-the-art methods consistently.

Towards interpretable learned representations for ecoacoustics using variational auto-encoding

Ecoacoustics is an emerging science that seeks to understand the role of sound in ecological processes. Passive acoustic monitoring is being used to collect vast quantities of soundscape audio recordings to study variations in acoustic community and monitor biodiversity. However, extracting relevant information from soundscape recordings is non-trivial. Recent approaches to machine-learned acoustic features appear promising but are limited by at least three issues: inductive biases, lack of interpretability and crude temporal integration. In this paper we introduce a novel self-supervised representation learning algorithm for ecoacoustics - a convolutional Variational Auto-encoder (VAE) - and directly address these shortcomings. Firstly, we train the network on soundscape recordings from temperate and tropical field sites along a gradient of ecological degradation to provide a more relevant inductive bias than prior approaches. Secondly, we present a new method that allows interpretation of the latent space for the first time, giving insight into the basis of classification. Thirdly, we advance existing methods for temporal aggregation of learned embeddings by encoding latent features as a distribution over time. Under our approach to increase interpretability, we provide insight into how learned features drive habitat classification for the first time: inspection of latent space confirms that varying combinations of biophony, geophony and anthrophony are used to infer sites along a degradation gradient. Our novel temporal encoding method increases sensitivity to periodic signals and improves on previous research that uses time-averaged representations for site classification. This approach also reveals the contribution of hardware-specific frequency response that create a potential bias; we demonstrate how a simple linear transformation can be used to mitigate the effect of hardware variance on the learned representation under our approach. Our novel approach paves the way for development of a new class of deep neural networks that afford more interpretable learned ecoacoustic representations to advance both fundamental and applied science and support global conservation efforts.

Discriminative Dictionary Pair Learning With Scale-Constrained Structured Representation for Image Classification.

The dictionary pair learning (DPL) model aims to design a synthesis dictionary and an analysis dictionary to accomplish the goal of rapid sample encoding. In this article, we propose a novel structured representation learning algorithm based on the DPL for image classification. It is referred to as discriminative DPL with scale-constrained structured representation (DPL-SCSR). The proposed DPL-SCSR utilizes the binary label matrix of dictionary atoms to project the representation into the corresponding label space of the training samples. By imposing a non-negative constraint, the learned representation adaptively approximates a block-diagonal structure. This innovative transformation is also capable of controlling the scale of the block-diagonal representation by enforcing the sum of within-class coefficients of each sample to 1, which means that the dictionary atoms of each class compete to represent the samples from the same class. This implies that the requirement of similarity preservation is considered from the perspective of the constraint on the sum of coefficients. More importantly, the DPL-SCSR does not need to design a classifier in the representation space as the label matrix of the dictionary can also be used as an efficient linear classifier. Finally, the DPL-SCSR imposes the l2,p -norm on the analysis dictionary to make the process of feature extraction more interpretable. The DPL-SCSR seamlessly incorporates the scale-constrained structured representation learning, within-class similarity preservation of representation, and the linear classifier into one regularization term, which dramatically reduces the complexity of training and parameter tuning. The experimental results on several popular image classification datasets show that our DPL-SCSR can deliver superior performance compared with the state-of-the-art (SOTA) dictionary learning methods. The MATLAB code of this article is available at https://github.com/chenzhe207/DPL-SCSR.

A Network Representation Learning Model Based on Multiple Remodeling of Node Attributes

Current network representation learning models mainly use matrix factorization-based and neural network-based approaches, and most models still focus only on local neighbor features of nodes. Knowledge representation learning aims to learn low-dimensional dense representations of entities and relations from structured knowledge graphs, and most models use the triplets to capture semantic, logical, and topological features between entities and relations. In order to extend the generalization capability of the network representation learning models, this paper proposes a network representation learning algorithm based on multiple remodeling of node attributes named MRNR. The model constructs the knowledge triplets through the textual association relationships between nodes. Meanwhile, a novel co-occurrence word training method has been proposed. Multiple remodeling of node attributes can significantly improve the effectiveness of network representation learning. At the same time, MRNR introduces the attention mechanism to achieve the weight information for key co-occurrence words and triplets, which further models the semantic and topological features between entities and relations, and it makes the network embedding more accurate and has better generalization ability.

AMGDTI: drug-target interaction prediction based on adaptive meta-graph learning in heterogeneous network.

Prediction of drug-target interactions (DTIs) is essential in medicine field, since it benefits the identification of molecular structures potentially interacting with drugs and facilitates the discovery and reposition of drugs. Recently, much attention has been attracted to network representation learning to learn rich information from heterogeneous data. Although network representation learning algorithms have achieved success in predicting DTI, several manually designed meta-graphs limit the capability of extracting complex semantic information. To address the problem, we introduce an adaptive meta-graph-based method, termed AMGDTI, for DTI prediction. In the proposed AMGDTI, the semantic information is automatically aggregated from a heterogeneous network by training an adaptive meta-graph, thereby achieving efficient information integration without requiring domain knowledge. The effectiveness of the proposed AMGDTI is verified on two benchmark datasets. Experimental results demonstrate that the AMGDTI method overall outperforms eight state-of-the-art methods in predicting DTI and achieves the accurate identification of novel DTIs. It is also verified that the adaptive meta-graph exhibits flexibility and effectively captures complex fine-grained semantic information, enabling the learning of intricate heterogeneous network topology and the inference of potential drug-target relationship.