Meaningful Representation Research Articles

Graph masked self-distillation learning for prediction of mutation impact on protein–protein interactions

Assessing mutation impact on the binding affinity change (ΔΔG) of protein–protein interactions (PPIs) plays a crucial role in unraveling structural-functional intricacies of proteins and developing innovative protein designs. In this study, we present a deep learning framework, PIANO, for improved prediction of ΔΔG in PPIs. The PIANO framework leverages a graph masked self-distillation scheme for protein structural geometric representation pre-training, which effectively captures the structural context representations surrounding mutation sites, and makes predictions using a multi-branch network consisting of multiple encoders for amino acids, atoms, and protein sequences. Extensive experiments demonstrated its superior prediction performance and the capability of pre-trained encoder in capturing meaningful representations. Compared to previous methods, PIANO can be widely applied on both holo complex structures and apo monomer structures. Moreover, we illustrated the practical applicability of PIANO in highlighting pathogenic mutations and crucial proteins, and distinguishing de novo mutations in disease cases and controls in PPI systems. Overall, PIANO offers a powerful deep learning tool, which may provide valuable insights into the study of drug design, therapeutic intervention, and protein engineering.

Domain embeddings for generating complex descriptions of concepts in Italian language.

In this work, we propose a Distributional Semantic resource enriched with linguistic and lexical information extracted from electronic dictionaries. This resource is designed to bridge the gap between the continuous semantic values represented by distributional vectors and the discrete descriptions provided by general semantics theory. Recently, many researchers have focused on the connection between embeddings and a comprehensive theory of semantics and meaning. This often involves translating the representation of word meanings in Distributional Models into a set of discrete, manually constructed properties, such as semantic primitives or features, using neural decoding techniques. Our approach introduces an alternative strategy based on linguistic data. We have developed a collection of domain-specific co-occurrence matrices derived from two sources: a list of Italian nouns classified into four semantic traits and 20 concrete noun sub-categories and Italian verbs classified by their semantic classes. In these matrices, the co-occurrence values for each word are calculated exclusively with a defined set of words relevant to a particular lexical domain. The resource includes 21 domain-specific matrices, one comprehensive matrix, and a Graphical User Interface. Our model facilitates the generation of reasoned semantic descriptions of concepts by selecting matrices directly associated with concrete conceptual knowledge, such as a matrix based on location nouns and the concept of animal habitats. We assessed the utility of the resource through two experiments, achieving promising outcomes in both the automatic classification of animal nouns and the extraction of animal features.

Representations of stimulus meaning in the hippocampus.

The ability to discriminate and categorize the meaning of environmental stimuli and respond accordingly is essential for survival. The ventral hippocampus (vHPC) controls emotional and motivated behaviors in response to environmental cues and is hypothesized to do so in part by deciphering the positive or negative quality of these cues. Yet, what features of the environment are represented in the activity patterns of vCA1 neurons, and whether the positive or negative meaning of a stimulus is present at this stage, remains unclear. Here, using 2-photon calcium imaging across six different experimental paradigms, we consistently found that vCA1 ensembles encode the identity, sensory features, and intensity of learned and innately salient stimuli, but not their overall valence. These results offer a reappraisal of vCA1 function, wherein information corresponding to individual stimulus features and their behavioral saliency predominates, while valence-related information is attached elsewhere.

Interpretable spatially aware dimension reduction of spatial transcriptomics with STAMP.

Spatial transcriptomics produces high-dimensional gene expression measurements with spatial context. Obtaining a biologically meaningful low-dimensional representation of such data is crucial for effective interpretation and downstream analysis. Here, we present Spatial Transcriptomics Analysis with topic Modeling to uncover spatial Patterns (STAMP), an interpretable spatially aware dimension reduction method built on a deep generative model that returns biologically relevant, low-dimensional spatial topics and associated gene modules. STAMP can analyze data ranging from a single section to multiple sections and from different technologies to time-series data, returning topics matching known biological domains and associated gene modules containing established markers highly ranked within. In a lung cancer sample, STAMP delineated cell states with supporting markers at a higher resolution than the original annotation and uncovered cancer-associated fibroblasts concentrated on the tumor edge's exterior. In time-series data of mouse embryonic development, STAMP disentangled the erythro-myeloid hematopoiesis and hepatocytes developmental trajectories within the liver. STAMP is highly scalable and can handle more than 500,000 cells.

Policy engagement as ‘empowered representation’: democratic mediation through a participatory research project on climate resilience

Background: The article analyses the policy engagement component of a research project on climate resilience in vulnerable communities that took place in Cape Town, South Africa. Conducted in 2022, the engagement included community and stakeholder events in three research sites, and a cross-cutting policy event with municipal officials, held at the end of the project. Importantly, this policy engagement process occurred in a context of political marginalisation, that is, one characterised by low trust, and little meaningful representation or even communication between these vulnerable communities and the city. Aims and objectives: This article examines the impact of policy engagement on political relations between local government and vulnerable communities. Methods: The overall methodology of the article is qualitative, using an illustrative case-study research design to unpack the subjective experiences of both government officials and residents of vulnerable communities. Primary data included many primary documents, direct observation of the engagements and post-event interviews. Findings: First, the engagement process created new ‘invented’ spaces for the representation of community perspectives to the city, and the city’s perspective to the community. Second, the engagement facilitated community self-representation through educating community members to advocate for their ideas in these new invented spaces. Third, this engagement tended to be more constructive and deliberative than polarising and confrontational. Discussion and conclusions: Drawing on the theoretical framework of ‘political mediation’, the policy engagement process is characterised as a positive instance of democratic mediation through ‘empowered representation’, with some specified limitations.

Understanding the structure of knowledge graphs with ABSTAT profiles

While there has been a trend in the last decades for publishing large-scale and highly-interconnected Knowledge Graphs (KGs), their users often get overwhelmed by the task of understanding their content as a result of their size and complexity. Data profiling approaches have been proposed to summarize large KGs into concise and meaningful representations, so that they can be better explored, processed, and managed. Profiles based on schema patterns represent each triple in a KG with its schema-level counterpart, thus covering the entire KG with profiles of considerable size. In this paper, we provide empirical evidence that profiles based on schema patterns, if explored with suitable mechanisms, can be useful to help users understand the content of big and complex KGs. ABSTAT provides concise pattern-based profiles and comes with faceted interfaces for profile exploration. Using this tool we present a user study based on query completion tasks. We demonstrate that users who look at ABSTAT profiles formulate their queries better and faster than users browsing the ontology of the KGs. The latter is a pretty strong baseline considering that many KGs do not even come with a specific ontology to be explored by the users. To the best of our knowledge, this is the first attempt to investigate the impact of profiling techniques on tasks related to knowledge graph understanding with a user study.

Spike-based computational primitives to reproduce neural dynamics in the parietal cortex during motor preparation

Abstract Biologically plausible spiking neural network models of sensory cortices can be instrumental in understanding and validating their principles of computation.
 Models based on Cortical Computational Primitives (CCPs), such as Hebbian plasticity and Winner-Take-All (WTA) networks, have already been successful in this approach.
 However, the specific nature and roles of CCPs in sensorimotor cortices during cognitive tasks are yet to be fully deciphered.
 The evolution of motor intention in the Posterior Parietal Cortex (PPC) before arm-reaching movements is a well-suited cognitive process to assess the effectiveness of different CCPs.
 To this end, we propose a biologically plausible model composed of heterogeneous spiking neurons which implements and combines multiple CCPs, such as multi-timescale learning and soft WTA modules.
 By training the model to replicate the dynamics of in-vivo recordings from non-human primates, we show how it is effective in generating meaningful representations from unbalanced input data, and in faithfully reproducing the transition from motor planning to action selection.
 Our findings elucidate the importance of distributing spike-based plasticity across multi-timescales, and provide an explanation for the role of different CCPs in models of frontoparietal cortical networks for performing multisensory integration to efficiently inform action execution.

Neural Semantic Parsing with Extremely Rich Symbolic Meaning Representations

Neural Semantic Parsing with Extremely Rich Symbolic Meaning Representations

Multi-adversarial autoencoders: Stable, faster and self-adaptive representation learning

The variational autoencoder (VAE) and generative adversarial networks (GAN) are two prominent approaches to achieving a probabilistic generative model by way of an autoencoder and a two-player minimax game. While VAEs often suffer from over-simplified posterior approximations, the adversarial autoencoder (AAE) has shown promise by adopting GAN to match the variational posterior to an arbitrary prior through adversarial training. Both VAEs and GANs face significant challenges such as training stability, mode collapse, and difficulty in extracting meaningful latent representations. In this paper, we propose the Multi-adversarial Autoencoder (MAAE), which extends the AAE framework by incorporating multiple discriminators and enabling soft-ensemble feedback. By adaptively regulating the collective feedback from multiple discriminators, MAAE captures a balance between fitting the data distribution and performing accurate inference and accelerates training stability while extracting meaningful and interpretable latent representations. Experimental evaluations on MNIST, CIFAR10, and CelebA datasets demonstrate significant improvements in latent representation, quality of generated samples, log-likelihood, and a pairwise comparison metric, with comparisons to recent methods.

TC-DTA: Predicting Drug-Target Binding Affinity With Transformer and Convolutional Neural Networks.

Bioinformatics is a rapidly evolving field that applies computational methods to analyze and interpret biological data. A key task in bioinformatics is identifying novel drug-target interactions (DTIs), which plays a crucial role in drug discovery. Most computational approaches treat DTI prediction as a binary classification problem, determining whether drug-target pairs interact. However, with the growing availability of drug-target binding affinity data, this binary task can be reframed as a regression problem focused on drug-target affinity (DTA). DTA quantifies the strength of drug-target binding, offering more detailed insights than DTI and serving as a valuable tool for virtual screening in drug discovery. Accurately predicting compound interactions with targets can accelerate the drug development process. In this study, we introduce a deep learning model named TC-DTA for DTA prediction, leveraging convolutional neural networks (CNN) and the encoder module of the transformer architecture. We begin by extracting raw drug SMILES strings and protein amino acid sequences from the dataset, which are then represented using various encoding methods. Subsequently, we employ CNN and the transformer's encoder module to extract features from the drug SMILES strings and protein sequences, respectively. Finally, the feature information is concatenated and input into a multi-layer perceptron to predict binding affinity scores. We evaluated our model on two benchmark DTA datasets, Davis and KIBA, comparing it with methods such as KronRLS, SimBoost, and DeepDTA. Our model, TC-DTA, outperformed these baseline methods based on evaluation metrics like Mean Squared Error (MSE), Concordance Index (CI), and Regression towards the Mean Index ( rm2 ). These results highlight the effectiveness of the Transformer's encoder and CNN in extracting meaningful representations from sequences, thereby enhancing DTA prediction accuracy. This deep learning model can accelerate drug discovery by identifying drug candidates with high binding affinity to specific targets. Compared to traditional methods, machine learning technology offers a more effective and efficient approach to drug discovery.

Refining Software Defect Prediction through Attentive Neural Models for Code Understanding

Identifying defects through manual software testing is a resource-intensive task in software development. To alleviate this, software defect prediction identifies code segments likely to contain faults using data-driven methods. Traditional techniques rely on static code metrics, which often fail to reflect the deeper syntactic and semantic features of the code. This paper introduces a novel framework that utilizes transformer-based networks with attention mechanisms to predict software defects. The framework encodes input vectors to develop meaningful representations of software modules. A bidirectional transformer encoder is employed to model programming languages, followed by fine-tuning with labeled data to detect defects. The performance of the framework is assessed through experiments across various software projects and compared against baseline techniques. Additionally, statistical hypothesis testing and an ablation study are performed to assess the impact of different parameter choices. The empirical findings indicate that the proposed approach can increase classification accuracy by an average of 15.93% and improve the F1 score by up to 44.26% compared to traditional methods.

Rethinking self-supervised learning for time series forecasting: A temporal perspective

Self-supervised learning has garnered significant attention for its ability to learn meaningful representations. Recent advancements have introduced self-supervised methods for time series forecasting. However, these efforts have faced limitations due to two primary drawbacks. Firstly, these approaches often borrow techniques from vision and language domains without adequately addressing the unique temporal dependencies inherent in time series data. Secondly, time series often show that the distribution shifts over time, which makes accurate forecasting challenging. In response to these issues, we propose TempSSL, a self-supervised learning framework designed for time series forecasting. TempSSL divides the time series data into context (history data) and target (future data), employing two pre-training strategies: (1) Temporal Masked Modeling (TMM) designed to capture temporal dependencies by reconstructing future time series based on historical context; (2) Temporal Contrastive Learning (TCL) employs context and target as positive samples to enhance discriminative representations and mitigate distribution shifts within the time series. TempSSL’s innovation lies in two key aspects. Firstly, it underscores the importance of temporal dependencies for time series forecasting by designing specific pre-training tasks. Secondly, it effectively integrates contrastive learning and masked modeling, leveraging their respective strengths to develop time series representation with strong instance discriminability and local perceptibility. Extensive experiments across seven widely used benchmark datasets demonstrate that TempSSL consistently outperforms existing self-supervised and end-to-end forecasting methods, achieving improvements ranging from 1.92% ∼ 78.12%. Additionally, TempSSL’s practical effectiveness is further demonstrated through successful application in natural gas demand forecasting.

Learning protein language contrastive models with multi-knowledge representation

Protein representation learning plays a crucial role in obtaining a comprehensive understanding of biological regulatory mechanisms and in developing proteins and drugs for therapeutic purposes. However, labeled proteins, such as sequenced and functionally annotated data, are incomplete and few. Thus, contrastive learning has emerged as the preferred technique for learning meaningful representations from unlabeled data samples. In addition, at present, natural proteins cannot be fully described by extracting protein knowledge from a single domain. Therefore, Pro-CoRL, a protein contrastive models framework based on multi-knowledge representation learning, was proposed in this study. In particular, Pro-CoRL smooths the objective function using convex approximation, thereby improving the stability of training. Extensive experiments on predicting protein–protein interaction types and clustering protein families have confirmed the high accuracy and robustness of Pro-CoRL.

Embeddings for Efficient Literature Screening: A Primer for Life Science Investigators

As the number of publications is quickly growing in any area of science, the need to efficiently find relevant information amidst a large number of similarly themed articles becomes very important. Semantic searching through text documents has the potential to overcome the limits of keyword-based searches, especially since the introduction of attention-based transformers, which can capture contextual nuances of meaning in single words, sentences, or whole documents. The deployment of these computational tools has been made simpler and accessible to investigators in every field of research thanks to a growing number of dedicated libraries, but knowledge of how meaning representation strategies work is crucial to making the most out of these instruments. The present work aims at introducing the technical evolution of the meaning representation systems, from vectors to embeddings and transformers tailored to life science investigators with no previous knowledge of natural language processing.

Do LSTM memory states reflect the relationships in reduced-complexity sandy shoreline models

Equilibrium-based models are a transparent method of modelling shoreline change, though often too simplistic to capture complex dynamics. Conversely, deep learning methodologies offer greater predictive power at the expense of transparency. In this research we scrutinize the internal workings of an LSTM shoreline model. A regression-based probe is used to show that cell state vectors, responsible for past-to-future information flow, autonomously generate equilibrium-like information akin to the physics-based equilibrium term of the ShoreFor model, Ωeq. The variation in probe skill throughout training is tracked to show that at 5 of 6 transects, the LSTM was able to meaningfully acquire equilibrium information (ΣΔR2 = 0.3–0.6). The results of this work offer evidence that an LSTM may model shoreline change with internal methods that are consistent with the current understanding of coastal shoreline dynamics. These physically meaningful representations emphasize the importance of co-evolution between machine learning and physics-based approaches moving forward.

Advancing New Governance Models for Gender Data in Climate Resilience Funding

Abstract Globally, between USD 850 to USD 940 billion was allocated for 2021 climate finance. However, a mere 0.01 percent of this was directed towards projects addressing both climate change and women’s rights. Unfortunately, the climate crisis is not gender-neutral. Yet, gender data analysis and the significant role of women in climate action are often overlooked, resulting in critical issues being underrepresented in climate budgets. In today’s data-driven world, gender data assumes a critical role in highlighting women’s significant contributions to climate change, thereby influencing funding decisions. Within this context, the Room 9 sub-group of the “17 Rooms initiative” is currently exploring the potential of data cooperatives to facilitate financing for women’s groups transitioning towards climate-resilient practices. This paper explores the intersection of gender data and climate resilience funding and advocates for improved governance models that ensure accurate, comprehensive, and meaningful representation of gender data in climate funding decision-making processes.

An Iteratively Reweighted Importance Kernel Bayesian Filtering Approach for High-Dimensional Data Processing

This paper proposes an iteratively re-weighted importance kernel Bayes filter (IRe-KBF) method for handling high-dimensional or complex data in Bayesian filtering problems. This innovative approach incorporates importance weights and an iterative re-weighting scheme inspired by iteratively re-weighted Least Squares (IRLS) to enhance the robustness and accuracy of Bayesian inference. The proposed method does not require explicit specification of prior and likelihood distributions; instead, it learns the kernel mean representations from training data. Experimental results demonstrate the superior performance of this method over traditional KBF methods on high-dimensional datasets.

SLBDetection-Net: Towards closed-set and open-set student learning behavior detection in smart classroom of K-12 education

Through effective analysis of K-12 education students’ learning behaviors in the classroom, it is possible to greatly improve the interaction between teaching and learning, thereby enhancing the quality of education. However, current traditional analysis of student classroom behavior focuses on closed-set behavior detection in a single scenario. For complex and open real classroom environments, the challenge lies in obtaining meaningful representations of behaviors in small and densely populated complex scenes, while also achieving good performance in both closed-set and open-set environments. To address these challenges, this study introduces a novel method for detecting student learning behavior in both closed-set and open-set scenarios, termed SLBDetection-Net. This method focuses on accurately capturing learning behavior representation, with a specific emphasis on Multi-scale Focusing Key Information (MFKI). The study begins with designing a Learning Behavior-aware Attention (LBA) mechanism, dedicated to extracting key features of learning behaviors and capturing the complex characteristics of targets across different scales. Building on this attention mechanism, a backbone network feature encoder, LBA-Swin Transformer Block, is constructed to form the comprehensive SLBDetection-Net. The effectiveness of SLBDetection-Net is validated through rigorous testing and evaluation on real classroom scenario data of K-12 education, comparing its performance with the State-of-the-Art (SOTA) methods. The results demonstrate that SLBDetection-Net achieves a mean Average Precision (mAP) of 96.4% on the ClaBehavior dataset and 85.9% on the SCB dataset. These findings underscore the method’s significant advantages in enhancing detection precision and efficiency in both closed-set and open-set scenarios, thereby expanding the application scope of educational assessment frameworks. The source code of this study is publicly available at https://github.com/CCNUZFW/SLBDetection-Net.

Molecular Classification with Graph ConvolutionalNetworks: Exploring the MUTAG Dataset for Mutagenicity Prediction

This paper presents the implementation of a Graph Convolutional Network (GCN) for the classification of chemical compounds using the MUTAG dataset, which consists of 188 ni- troaromatic compounds labeled according to their mutagenicity. The GCN model leverages the inherent graph structure of molec-ular data to capture and learn from the relationships between atoms and bonds, represented as nodes and edges, respectively. By utilizing three graph convolutional layers followed by a global mean pooling layer, the model effectively aggregates node features to generate meaningful graph-level representations. The model was trained using the Adam optimizer with a learning rate of 0.01, and cross-entropy loss was employed to supervise the classification task. The results demonstrate the efficacy of GCNs in graph classification tasks, with the model achieving a training accuracy of 79.33% and a test accuracy of 76.32%. This study highlights the potential of GCNs in cheminformatics and other domains where graph-structured data is prevalent, paving the way for further exploration and application of advanced graph neural networks in similar tasks.

A New Version of a Broadly Applicable, Cross-lingual Meaning Representation Formalism and Its Significance for Biomedical Sciences

A New Version of a Broadly Applicable, Cross-lingual Meaning Representation Formalism and Its Significance for Biomedical Sciences