Meaningful Representation Research Articles

Empowering sign language communication: Integrating sentiment and semantics for facial expression synthesis

Translating written sentences from oral languages to a sequence of manual and non-manual gestures plays a crucial role in building a more inclusive society for deaf and hard-of-hearing people. Facial expressions (non-manual), in particular, are responsible for encoding the grammar of the sentence to be spoken, applying punctuation, pronouns, or emphasizing signs. These non-manual gestures are closely related to the semantics of the sentence being spoken and also to the utterance of the speaker’s emotions. However, most Sign Language Production (SLP) approaches are centered on synthesizing manual gestures and do not focus on modeling the speaker’s expression. This paper introduces a new method focused in synthesizing facial expressions for sign language. Our goal is to improve sign language production by integrating sentiment information in facial expression generation. The approach leverages a sentence’s sentiment and semantic features to sample from a meaningful representation space, integrating the bias of the non-manual components into the sign language production process. To evaluate our method, we extend the Fréchet gesture distance (FGD) and propose a new metric called Fréchet Expression Distance (FED) and apply an extensive set of metrics to assess the quality of specific regions of the face. The experimental results showed that our method achieved state of the art, being superior to the competitors on How2Sign and PHOENIX14T datasets. Moreover, our architecture is based on a carefully designed graph pyramid that makes it simpler, easier to train, and capable of leveraging emotions to produce facial expressions. Our code and pretrained models will be available at: https://github.com/verlab/empowering-sign-language.

Basic Polarimetric Characteristics of Full-Polarization Chandrayaan-II DFSAR Data Analyzed Using MIDAS (Microwave Data Analysis Software)

This study focuses on the analysis of basic polarimetric characteristics of full-polarization DFSAR data specifically through image formats using the Microwave Data Analysis Software (MIDAS). The research emphasizes visual representation, leveraging image-based outputs to interpret polarimetric parameters. These image formats provide a clearer understanding of surface scattering mechanisms and enhance the ability to perform detailed environmental and geological assessments using radar data. The study demonstrates how MIDAS can effectively process DFSAR data into meaningful visual representations for advanced analysis.

Learning meaningful representation of single-neuron morphology via large-scale pre-training.

Single-neuron morphology, the study of the structure, form, and shape of a group of specialized cells in the nervous system, is of vital importance to define the type of neurons, assess changes in neuronal development and aging and determine the effects of brain disorders and treatments. Despite the recent surge in the amount of available neuron morphology reconstructions due to advancements in microscopy imaging, existing computational and deep learning methods for modeling neuron morphology have been limited in both scale and accuracy. In this paper, we propose MorphRep, a model for learning meaningful representation of neuron morphology pre-trained with over 250 000 existing neuron morphology data. By encoding the neuron morphology into graph-structured data, using graph transformers for feature encoding and enforcing the consistency between multiple augmented views of neuron morphology, MorphRep achieves the state of the art performance on widely used benchmarking datasets. Meanwhile, MorphRep can accurately characterize the neuron morphology space across neuron morphometrics, fine-grained cell types, brain regions and ages. Furthermore, MorphRep can be applied to distinguish neurons under a wide range of conditions, including genetic perturbation, drug injection, environment change and disease. In summary, MorphRep provides an effective strategy to embed and represent neuron morphology and can be a valuable tool in integrating cell morphology into single-cell multiomics analysis. The codebase has been deposited in https://github.com/YaxuanLi-cn/MorphRep.

Imputation of cancer proteomics data with a deep model that learns from many datasets.

Missing values are a major challenge in the analysis of mass spectrometry proteomics data. Missing values hinder reproducibility, decrease statistical power for identifying differentially expressed (DE) proteins and make it challenging to analyze low-abundance proteins. We present Lupine, a deep learning-based method for imputing, or estimating, missing values in tandem mass tag (TMT) proteomics data. Lupine is, to our knowledge, the first imputation method that is designed to learn jointly from many datasets, and we provide evidence that this approach leads to more accurate predictions. We validated Lupine by applying it to TMT data from >1,000 cancer patient samples spanning ten cancer types from the Clinical Proteomics Tumor Atlas Consortium (CPTAC). Lupine outperforms the state of the art for TMT imputation, identifies more DE proteins than other methods, corrects for TMT batch effects, and learns a meaningful representation of proteins and patient samples. Lupine is implemented as an open source Python package.

Improving deep representation learning via auxiliary learnable target coding

Deep representation learning is a subfield of machine learning that focuses on learning meaningful and useful representations of data through deep neural networks. However, existing methods for semantic classification typically employ pre-defined target codes such as the one-hot and the Hadamard codes, which can either fail or be less flexible to model inter-class correlation. In light of this, this paper introduces a novel learnable target coding as an auxiliary regularization of deep representation learning, which can not only incorporate latent dependency across classes but also impose geometric properties of target codes into representation space. Specifically, a margin-based triplet loss and a correlation consistency loss on the proposed target codes are designed to encourage more discriminative representations owing to enlarging between-class margins in representation space and favoring equal semantic correlation of learnable target codes respectively. Experimental results on several popular visual classification and retrieval benchmarks can demonstrate the effectiveness of our method on improving representation learning, especially for imbalanced data.

ORASIS-MAE Harnesses the Potential of Self-Learning from Partially Annotated Clinical Eye Movement Records

Self-supervised learning (SSL) has gained significant attention in the past decade for its capacity to utilize non-annotated datasets to learn meaningful data representations. In the medical domain, the challenge of constructing large annotated datasets presents a significant limitation, rendering SSL an ideal approach to address this constraint. In this study, we introduce a novel pretext task tailored to stimulus-driven eye movement data, along with a denoising task to improve the robustness against simulated eye tracking failures. Our proposed task aims to capture both the characteristics of the pilot (brain) and the motor (eye) by learning to reconstruct the eye movement position signal using up to 12.5% of the unmasked eye movement signal patches, along with the entire REMOBI target signal. Thus, the encoder learns a high-dimensional representation using a multivariate time series of length 8192 points, corresponding to approximately 40 s. We evaluate the learned representation on screening eight distinct groups of pathologies, including dyslexia, reading disorder, and attention deficit disorder, across four datasets of varying complexity and size. Furthermore, we explore various head architecture designs along with different transfer learning methods, demonstrating promising results with improvements of up to approximately 15%, leading to an overall macro F1 score of 61% and 61.5% on the Saccade and the Vergence datasets, respectively. Notably, our method achieves macro F1 scores of 64.7%, 66.1%, and 61.1% for screening dyslexia, reading disorder, and attention deficit disorder, respectively, on clinical data. These findings underscore the potential of self-learning algorithms in pathology screening, particularly in domains involving complex data such as stimulus-driven eye movement analysis.

Latent disentanglement in mesh variational autoencoders improves the diagnosis of craniofacial syndromes and aids surgical planning

Background and objective:The use of deep learning to undertake shape analysis of the complexities of the human head holds great promise. However, there have traditionally been a number of barriers to accurate modelling, especially when operating on both a global and local level. Methods:In this work, we will discuss the application of the Swap Disentangled Variational Autoencoder (SD-VAE) with relevance to Crouzon, Apert and Muenke syndromes. The model is trained on a dataset of 3D meshes of healthy and syndromic patients which was increased in size with a novel data augmentation technique based on spectral interpolation. Thanks to its semantically meaningful and disentangled latent representation, SD-VAE is used to analyse and generate head shapes while considering the influence of different anatomical sub-units. Results:Although syndrome classification is performed on the entire mesh, it is also possible, for the first time, to analyse the influence of each region of the head on the syndromic phenotype. By manipulating specific parameters of the generative model, and producing procedure-specific new shapes, it is also possible to approximate the outcome of a range of craniofacial surgical procedures. Conclusion:This work opens new avenues to advance diagnosis, aids surgical planning and allows for the objective evaluation of surgical outcomes. Our code is available at github.com/simofoti/CraniofacialSD-VAE.

Document-Level Event Argument Extraction with Sparse Representation Attention

Document-level Event Argument Extraction (DEAE) aims to extract structural event knowledge composed of arguments and roles beyond the sentence level. Existing methods mainly focus on designing prompts and using Abstract Meaning Representation (AMR) graph structure as additional features to enrich event argument representation. However, two challenges still remain: (1) the long-range dependency between event trigger and event arguments and (2) the distracting context in the document towards an event that can mislead the argument classification. To address these issues, we propose a novel document-level event argument extraction model named AMR Parser and Sparse Representation (APSR). Specifically, APSR sets inter- and intra-sentential encoders to capture the contextual information in different scopes. Especially, in the intra-sentential encoder, APSR designs three types of sparse event argument attention mechanisms to extract the long-range dependency. Then, APSR constructs AMR semantic graphs, which capture the interactions among concepts well. Finally, APSR fuses the inter- and intra-sentential representations and predicts what role a candidate span plays. Experimental results on the RAMS and WikiEvents datasets demonstrate that APSR achieves a superior performance compared with competitive baselines in terms of F1 by 1.27% and 3.12%, respectively.

Manipulating and measuring variation in deep neural network (DNN) representations of objects

We explore how DNNs can be used to develop a computational understanding of individual differences in high-level visual cognition given their ability to generate rich meaningful object representations informed by their architecture, experience, and training protocols. As a first step to quantifying individual differences in DNN representations, we systematically explored the robustness of a variety of representational similarity measures: Representational Similarity Analysis (RSA), Centered Kernel Alignment (CKA), and Projection-Weighted Canonical Correlation Analysis (PWCCA), with an eye to how these measures are used in cognitive science, cognitive neuroscience, and vision science. To manipulate object representations, we next created a large set of models varying in random initial weights and random training image order, training image frequencies, training category frequencies, and model size and architecture and measured the representational variation caused by each manipulation. We examined both small (All-CNN-C) and commonly-used large (VGG and ResNet) DNN architectures. To provide a comparison for the magnitude of representational differences, we established a baseline based on the representational variation caused by image-augmentation techniques used to train those DNNs. We found that variation in model randomization and model size never exceeded baseline. By contrast, differences in training image frequency and training category frequencies caused representational variation that exceeded baseline, with training category frequency manipulations exceeding baseline earlier in the networks. These findings provide insights into the magnitude of representational variations that can be expected with a range of manipulations and provide a springboard for further exploration of systematic model variations aimed at modeling individual differences in high-level visual cognition.

Mondrian Abstraction and Language Model Embeddings for Differential Pathway Analysis.

In this study, we introduce the Mondrian Map, an innovative visualization tool inspired by Piet Mondrian's abstract art, to address the complexities inherent in visualizing biological networks. By converting intricate biological data into a structured and intuitive format, the Mondrian Map enables clear and meaningful representations of biological pathways, facilitating a deeper understanding of molecular dynamics. Each pathway is represented by a square whose size corresponds to fold change, with color indicating the direction of regulation (up or down) and statistical significance. The spatial arrangement of pathways is derived from language model embeddings, preserving neighborhood relationships and enabling the identification of clusters of related pathways. Additionally, colored lines highlight potential crosstalk between pathways, with distinctions between short- and long-range functional interactions. In a case study of glioblastoma multiforme (GBM), the Mondrian Map effectively revealed distinct pathway patterns across patient profiles at different stages of disease progression. These insights demonstrate the tool's potential to enhance downstream bioinformatics analysis by providing a more comprehensive and visually accessible overview of pathway interactions, offering new avenues for therapeutic exploration and personalized medicine.

Investigating Contrastive Pair Learning's Frontiers in Supervised, Semisupervised, and Self-Supervised Learning.

In recent years, contrastive learning has been a highly favored method for self-supervised representation learning, which significantly improves the unsupervised training of deep image models. Self-supervised learning is a subset of unsupervised learning in which the learning process is supervised by creating pseudolabels from the data themselves. Using supervised final adjustments after unsupervised pretraining is one way to take the most valuable information from a vast collection of unlabeled data and teach from a small number of labeled instances. This study aims firstly to compare contrastive learning with other traditional learning models; secondly to demonstrate by experimental studies the superiority of contrastive learning during classification; thirdly to fine-tune performance using pretrained models and appropriate hyperparameter selection; and finally to address the challenge of using contrastive learning techniques to produce data representations with semantic meaning that are independent of irrelevant factors like position, lighting, and background. Relying on contrastive techniques, the model efficiently captures meaningful representations by discerning similarities and differences between modified copies of the same image. The proposed strategy, involving unsupervised pretraining followed by supervised fine-tuning, improves the robustness, accuracy, and knowledge extraction of deep image models. The results show that even with a modest 5% of data labeled, the semisupervised model achieves an accuracy of 57.72%. However, the use of supervised learning with a contrastive approach and careful hyperparameter tuning increases accuracy to 85.43%. Further adjustment of the hyperparameters resulted in an excellent accuracy of 88.70%.

EEG Cortical Sources of Theta, Beta and Gamma Frequencies Orchestrate Audio–Visual Interactions

Background The fascinating ability of brain to integrate information from multiple sensory inputs has intrigued many researchers. Audio–visual (AV) interaction is a form of multisensory integration which we encounter to form meaningful representations of the environment around us. There is limited literature related to the underlying neural mechanisms. Purpose Quantitative EEG (QEEG), a tool with high temporal resolution can be used to understand cortical sources of AV interactions. Methods EEG data was recorded using 128 channels from 30 healthy subjects using audio, visual and AV stimuli in ‘object detection task’. Electrical source imaging was performed using s-LORETA across seven frequency bands (lower alpha 1, lower alpha 2, upper alpha, beta, delta, gamma, theta) during AV versus unimodal conditions across 66 gyri. Results The cortical sources were activated in the theta, beta, and gamma bands in cross modal versus unimodal conditions, which we propose, reflect neural communication for AV interaction network. The cortical sources constituted areas involved with visual processing, auditory processing, established multisensory (frontotemporal cortex, parietal cortex, middle temporal gyrus, superior frontal gyrus, inferior frontal gyrus, precentral gyrus) and potential multisensory areas (paracentral, postcentral and subcallosal). Conclusion Together, these results offer an integrative view of cortical areas in frequency oscillations during AV interactions.

Gender Aspect of Media Discourse in Socialist Yugoslavia – Representation of Gender in the Women's Magazine "Bazar"

The paper revises the gender aspect of media discourse in socialist Yugoslavia on the example of one of the oldest and most popular Yugoslav women's magazine "Bazar." An analysis of the textual and visual content of the 1973 edition (issues 208–233) reaffirmed the assumption that the mass media – including The Women's Press – actively participate in the construction and reproduction of both "femininity" and "masculinity", and in the depiction of desirable/expected gender relations in a certain socio-historical moment (Yugoslav socialism). In addition to the main female character – a beautiful, young, modern and well-groomed woman who is successful in all fields (at work, at home, as a wife, mother, sister, friend, worker...) – there is also an equally caring and dedicated husband and father, a modern man, successful in his field of expertise. In other words, of all the roles that Yugoslav society has assigned to women/men, the following seem to be dominant: She is primarily 1) a wife, mother and housewife, 2) a hard worker, and 3) a woman who is clearly positioned in the new consumer society, while He is 1) Pater Familias, 2) a successful businessman, and 3) a modern man who keeps up with the times. The research has shown that these (and such) "Bazar" images of a woman / man are highly harmonized with the official representation of gender in the socialist Yugoslav society – She is the embodiment of the socialist "super-woman", while He is depicted as part of the hegemonic model of masculinity. The female-male relationship is presented in the experimental media discourse in two ways. The first (socialist) model of gender polarity – the one that is in the central media plan / easily observable / clearly legible – is placed in the official narrative framework of socialism as an ideology that follows the principle of equality as the dominant matrix in social/gender/economic/family relations. In the second narrative plan, there is a traditional model of gender polarity, which is based on binary oppositions – weak-strong, gentle-rough, subordinate-dominant, dependent-independent, right-wrong – performed by the female-male couple. By analyzing the media discourse of the experimental socialist periodical, it has also been established that the narrative of one of the most popular Yugoslav women's magazines contains all three key terms (for gender reading or in the media text) – these are power, gender (in)equality and representation of meaning. Interpreting textual and visual content as a system of representation (and not presentation) of reality, it becomes obvious that despite the change in social paradigm in the years after World War II, the policy of gender representation in the context of power relations (man over woman) continued, which Yugoslav socialism – as a doctrine of equality – clearly could not (or did not want to) eradicate in practice.

Contrastive learning with hard negative samples for chest X-ray multi-label classification

Contrastive learning has gained significant popularity and achieved remarkable success in learning meaningful representations in various domains. This study addresses the significant problem of dependency on labeled data in chest radiography (CXR) images, which are crucial for diagnosing respiratory diseases such as pneumonia but are both time-consuming and expensive to annotate. Despite extensive research, existing studies on CXR largely depend on labeled data. To overcome this challenge, we propose a framework named SURE (similarity, uncertainty, and representativeness) for multi-label classification with hard negatives in contrastive learning. The proposed framework incorporates these aspects when handling hard negatives and effectively combines contrastive learning and downstream tasks for robust representation learning and multi-label classification. Experimental validation using three distinct CXR datasets demonstrates that our approach significantly reduces the dependency on labeled data while achieving notable performance improvements over existing methods, highlighting its potential effectiveness and efficiency in the CXR domain.

Translation as retelling

Abstract Taking the perspective that translation is a form of (re)narration that participates in (re)constructing social reality (Baker 2018), this article examines how translation is used as a rigorous means for mediating meaning, narrativity and cultural representation. Focusing on a case study of Amores do Céu e da Terra: Contos de Macau (2014), a Portuguese version of the original Chinese prose essay collection Youqing Tiandi (1991), this article examines the Portuguese version as “reconto” (retelling) to see how the public (collective) narratives of Macao are reframed by the translators in the target language and culture. The analysis reveals that the original narratives are reconstructed by the Portuguese translators through their agency at the contextual, textual, intertextual and paratextual levels. Such reframing has resulted in a new version that reconstructs the public narratives and cultural image of Macao for Portuguese readers.

Variation-aware directed graph convolutional networks for skeleton-based action recognition

Directed Graph convolutional networks (DGCNs) have been indeed gaining attention and being applied in skeleton-based action recognition tasks to capture the hierarchical relationships of skeleton via directed graph topology. However, they typically pay the same attention to regions with variations and regions with relative statics of the skeleton, which leads to low accuracy when recognizing fine-grained actions. To this end, we design an innovative variation-aware directed graph convolutional network (VA-DGCN) to focus on the regions where the variation takes place. VA-DGCN comprises a variation-aware directed spatial convolution (VDSC) module and a multi-scale contrastive temporal convolution (MCTC) module. Specifically, in order to capture subtle variations in fine-grained actions, VDSC introduce the average posture of the action sequence as the static anchor. In VDSC, the channel-specific topology branch is designed to model the different kinematic properties of different channels to extract global features, to which global-attention graph convolution is added for unnaturally connected joints. Subsequently, we identify the regions with variations by comparing global features acquired from action sequence and average posture. In temporal, MCTC comprises multiple branches for extracting temporal features at different scales. Moreover, to maximize the mutual information between branches, we introduce contrastive learning to drive the module to learn more meaningful action representations. We conduct extensive experiments on three public datasets to validate the feasibility and efficacy of our proposed VA-DGCN.

CliqueFluxNet: Unveiling EHR Insights with Stochastic Edge Fluxing and Maximal Clique Utilisation Using Graph Neural Networks

Electronic Health Records (EHRs) play a crucial role in shaping predictive are models, yet they encounter challenges such as significant data gaps and class imbalances. Traditional Graph Neural Network (GNN) approaches have limitations in fully leveraging neighbourhood data or demanding intensive computational requirements for regularisation. To address this challenge, we introduce CliqueFluxNet, a novel framework that innovatively constructs a patient similarity graph to maximise cliques, thereby highlighting strong inter-patient connections. At the heart of CliqueFluxNet lies its stochastic edge fluxing strategy — a dynamic process involving random edge addition and removal during training. This strategy aims to enhance the model’s generalisability and mitigate overfitting. Our empirical analysis, conducted on MIMIC-III and eICU datasets, focuses on the tasks of mortality and readmission prediction. It demonstrates significant progress in representation learning, particularly in scenarios with limited data availability. Qualitative assessments further underscore CliqueFluxNet’s effectiveness in extracting meaningful EHR representations, solidifying its potential for advancing GNN applications in healthcare analytics.

Exploring galaxy properties of eCALIFA with contrastive learning

Contrastive learning (CL) has emerged as a potent tool for building meaningful latent representations of galaxy properties across a broad spectrum of wavelengths, ranging from optical and infrared to radio frequencies. These latent representations facilitate a variety of downstream tasks, including galaxy classification, similarity searches in extensive datasets, and parameter estimation, which is why they are often referred to as foundation models for galaxies. In this study, we employ CL on the latest extended data release from the Calar Alto Legacy Integral Field Area (CALIFA) survey, which encompasses a total of 895 galaxies with enhanced spatial resolution that reaches the limits imposed by natural seeing (FWHMPSF ∼ 1.5). We demonstrate that CL can be effectively applied to Integral Field Unit (IFU) surveys, even with relatively small training sets, to construct meaningful embedding where galaxies are well separated based on their physical properties. We discover that the strongest correlations in the embedding space are observed with the equivalent width of Hα, galaxy morphology, stellar metallicity, luminosity-weighted age, stellar surface mass density, the [NII]/Hα ratio, and stellar mass, in descending order of correlation strength. Additionally, we illustrate the feasibility of unsupervised separation of galaxy populations along the star formation main sequence, successfully identifying the blue cloud and the red sequence in a two-cluster scenario, and the green valley population in a three-cluster scenario. Our findings indicate that galaxy luminosity profiles have minimal impact on the construction of the embedding space, suggesting that morphology and spectral features play a more significant role in distinguishing between galaxy populations. Moreover, we explore the use of CL for detecting variations in galaxy population distributions across different large-scale structures, including voids, clusters, and filaments and walls. Nonetheless, we acknowledge the limitations of the CL framework and our specific training set in detecting subtle differences in galaxy properties, such as the presence of an AGN or other minor scale variations that exceed the scope of primary parameters such as the stellar mass or morphology. Conclusively, we propose that CL can serve as an embedding function for the development of larger models capable of integrating data from multiple datasets, thereby advancing the construction of more comprehensive foundation models for galaxies.

Multimodality in the television advertisement of Raha premium maize meal

In language and communication, multimodality refers to using various modes of signification to convey meaning. These modes transcend language to engage with audiences at multiple sensory levels. Over time, communication has evolved to reflect a paradigm shift towards a multimodal perspective. This leads to a complex interplay between different modes of shaping interpretations and constructing meaning in discourse, which has attracted little research. This paper explores the concept of multimodality as used in the television advertisement discourse of Raha premium maize meal, using the Multimodal Discourse Analysis Theory. The objective is to investigate how viewers interpret and perceive different modes of signification employed in the advertisement. A descriptive research design with a qualitative approach is used in the collection and analysis of data. The findings reveal that each mode of signification has a unique role, some of which television viewers hardly denote their meaning. Further, politics, religion, and culture influence the perception of some modes of signification used in the advertisement. We recommend that the advertisements be tailored with modes of signification that resonate with the expectations of television viewers. The findings of this paper are intended to enrich linguistic theory concerning the analysis of the representation of meaning in a text and perceptions of modes of signification used.

Abstractive text summarization: State of the art, challenges, and improvements

Specifically focusing on the landscape of abstractive text summarization, as opposed to extractive techniques, this survey presents a comprehensive overview, delving into state-of-the-art techniques, prevailing challenges, and prospective research directions. We categorize the techniques into traditional sequence-to-sequence models, pre-trained large language models, reinforcement learning, hierarchical methods, and multi-modal summarization. Unlike prior works that did not examine complexities, scalability and comparisons of techniques in detail, this review takes a comprehensive approach encompassing state-of-the-art methods, challenges, solutions, comparisons, limitations and charts out future improvements — providing researchers an extensive overview to advance abstractive summarization research. We provide vital comparison tables across techniques categorized — offering insights into model complexity, scalability and appropriate applications. The paper highlights challenges such as inadequate meaning representation, factual consistency, controllable text summarization, cross-lingual summarization, and evaluation metrics, among others. Solutions leveraging knowledge incorporation and other innovative strategies are proposed to address these challenges. The paper concludes by highlighting emerging research areas like factual inconsistency, domain-specific, cross-lingual, multilingual, and long-document summarization, as well as handling noisy data. Our objective is to provide researchers and practitioners with a structured overview of the domain, enabling them to better understand the current landscape and identify potential areas for further research and improvement.