Text Representation Research Articles

A Hybrid Semantic Representation Method Based on Fusion Conceptual Knowledge and Weighted Word Embeddings for English Texts

The accuracy of traditional topic models may be compromised due to the sparsity of co-occurring vocabulary in the corpus, whereas conventional word embedding models tend to excessively prioritize contextual semantic information and inadequately capture domain-specific features in the text. This paper proposes a hybrid semantic representation method that combines a topic model that integrates conceptual knowledge with a weighted word embedding model. Specifically, we construct a topic model incorporating the Probase concept knowledge base to perform topic clustering and obtain topic semantic representation. Additionally, we design a weighted word embedding model to enhance the contextual semantic information representation of the text. The feature-based information fusion model is employed to integrate the two textual representations and generate a hybrid semantic representation. The hybrid semantic representation model proposed in this study was evaluated based on various English composition test sets. The findings demonstrate that the model presented in this paper exhibits superior accuracy and practical value compared to existing text representation methods.

A Picture May Be Worth a Hundred Words for Visual Question Answering

How far can textual representations go in understanding images? In image understanding, effective representations are essential. Deep visual features from object recognition models currently dominate various tasks, especially Visual Question Answering (VQA). However, these conventional features often struggle to capture image details in ways that match human understanding, and their decision processes lack interpretability. Meanwhile, the recent progress in language models suggests that descriptive text could offer a viable alternative. This paper investigated the use of descriptive text as an alternative to deep visual features in VQA. We propose to process description–question pairs rather than visual features, utilizing a language-only Transformer model. We also explored data augmentation strategies to enhance training set diversity and mitigate statistical bias. Extensive evaluation shows that textual representations using approximately a hundred words can effectively compete with deep visual features on both the VQA 2.0 and VQA-CP v2 datasets. Our qualitative experiments further reveal that these textual representations enable clearer investigation of VQA model decision processes, thereby improving interpretability.

Composite Hedges Nanopores codec system for rapid and portable DNA data readout with high INDEL-Correction

Reading digital information from highly dense but lightweight DNA medium nowadays relies on time-consuming next-generation sequencing. Nanopore sequencing holds the promise to overcome the efficiency problem, but high indel error rates lead to the requirement of large amount of high quality data for accurate readout. Here we introduce Composite Hedges Nanopores, capable of handling indel rates up to 15.9% and substitution rates up to 7.8%. The overall information density can be doubled from 0.59 to 1.17 by utilizing a degenerated eight-letter alphabet. We demonstrate that sequencing times of 20 and 120 minutes are sufficient for processing representative text and image files, respectively. Moreover, to achieve complete data recovery, it is estimated that text and image data require 4× and 8× physical redundancy of composite strands, respectively. Our codec system excels on both molecular design and equalized dictionary usage, laying a solid foundation approaching to real-time DNA data retrieval and encoding.

How do Large Language Models understand Genes and Cells

Researching genes and their interactions is crucial for deciphering the fundamental laws of cellular activity, advancing disease treatment, drug discovery, and more. Large language Models (LLMs), with their profound text comprehension and generation capabilities, have made significant strides across various natural science fields. However, their application in cell biology remains limited and a systematic evaluation of their performance is lacking. To address this gap, in this paper, we select seven mainstream LLMs and evaluate their performance across nine gene-related problem scenarios. Our findings indicate that LLMs possess a certain level of understanding of genes and cells, but still lag behind domain-specific models in comprehending transcriptional expression profiles. Moreover, we have improved the current method of textual representation of cells, enhancing the LLMs’ ability to tackle cell annotation tasks. We encourage cell biology researchers to leverage LLMs for problem-solving while being mindful of the associated challenges. We release our code and data at https://github.com/epang-ucas/Evaluate_LLMs_to_Genes .

Towards AI-Generated Essay Classification Using Numerical Text Representation

The detection of essays written by AI compared to those authored by students is increasingly becoming a significant issue in educational settings. This research examines various numerical text representation techniques to improve the classification of these essays. Utilizing a diverse dataset, we undertook several preprocessing steps, including data cleaning, tokenization, and lemmatization. Our system analyzes different text representation methods such as Bag of Words, TF-IDF, and fastText embeddings in conjunction with multiple classifiers. Our experiments showed that TF-IDF weights paired with logistic regression reached the highest accuracy of 99.82%. Methods like Bag of Words, TF-IDF, and fastText embeddings achieved accuracies exceeding 96.50% across all tested classifiers. Sentence embeddings, including MiniLM and distilBERT, yielded accuracies from 93.78% to 96.63%, indicating room for further refinement. Conversely, pre-trained fastText embeddings showed reduced performance, with a lowest accuracy of 89.88% in logistic regression. Remarkably, the XGBoost classifier delivered the highest minimum accuracy of 96.24%. Specificity and precision were above 99% for most methods, showcasing high capability in differentiating between student-created and AI-generated texts. This study underscores the vital role of choosing dataset-specific text representations to boost classification accuracy.

Learning and semiautomatic intention labeling for classification models: a COVID-19 dialog attendance study for chatbots

Abstract It is increasingly common to use chatbots as an interface to services. One of the main components of a chatbot is the Natural Language Understanding (NLU) model, which is responsible for interpreting the text and extracting the intent and entities present in that text. It’s possible to focus only on one of these tasks of NLU, such as intent classification. To train an NLU intent classification model, it’s generally necessary to use a considerable amount of annotated data, where each sentence of the dataset receives a label indicating an intent. Performing manually labeling data is arduous and impracticable, depending on the data volume. Thus, an unsupervised machine learning technique, such as data clustering, could be applied to find and label patterns in the data. For this task, it is essential to have an effective vector embedding representation of texts that depicts the semantic information and helps the machine understand the context, intent, and other nuances of the entire text. This paper extensively evaluates different text embedding models for clustering and labeling. We also apply some operations to improve the dataset’s quality, such as removing sentences and establishing various strategies for distance thresholds (cosine similarity) for the clusters’ centroids. Then, we trained some intent classification Models with two different architectures, one built with the Rasa framework and the other with a neural network (NN) using the attendance text from the Coronavirus Platform Service of Ceará, Brazil. We also manually annotated a dataset to be used as validation data. We conducted a study on semiautomatic labeling, implemented through clustering and visual inspection, which introduced some labeling errors to the intent classification models. However, it would be unfeasible to annotate the entire dataset manually. Nevertheless, results of competitive accuracy were still achieved with the trained models.

TL-CStrans Net: a vision robot for table tennis player action recognition driven via CS-Transformer.

Currently, the application of robotics technology in sports training and competitions is rapidly increasing. Traditional methods mainly rely on image or video data, neglecting the effective utilization of textual information. To address this issue, we propose: TL-CStrans Net: A vision robot for table tennis player action recognition driven via CS-Transformer. This is a multimodal approach that combines CS-Transformer, CLIP, and transfer learning techniques to effectively integrate visual and textual information. Firstly, we employ the CS-Transformer model as the neural computing backbone. By utilizing the CS-Transformer, we can effectively process visual information extracted from table tennis game scenes, enabling accurate stroke recognition. Then, we introduce the CLIP model, which combines computer vision and natural language processing. CLIP allows us to jointly learn representations of images and text, thereby aligning the visual and textual modalities. Finally, to reduce training and computational requirements, we leverage pre-trained CS-Transformer and CLIP models through transfer learning, which have already acquired knowledge from relevant domains, and apply them to table tennis stroke recognition tasks. Experimental results demonstrate the outstanding performance of TL-CStrans Net in table tennis stroke recognition. Our research is of significant importance in promoting the application of multimodal robotics technology in the field of sports and bridging the gap between neural computing, computer vision, and neuroscience.

Text-and-Image Learning Transformer for Cross-modal Person Re-identification

Text-based person re-identification aims to find the target person from a large pedestrian gallery with the given natural language description. Previous works mainly focus on embedding salient textual and visual representations in a common latent space by utilizing the dual-path structure or parameter-shared network. However, they still lack the ability to effectively extract fine-grained unimodal features as well as fuse the cross-modal data, leading to the increase of misaligned cases. To settle these issues, we propose a text-and-image implicit learning Transformer (TILT) to eliminate textual anisotropy and enhance the cross-modal alignment from both domains based on the bi-direction multimodal encoders. Specifically, we apply the pre-trained multimodal embedding module to overcome the unimodal anisotropy problem with contrastive learning, and map fine-grained features with dual encoder in bidirectional masking. Then, we design the cross-modal interaction encoder to comprehensively mine implicit cross-modal relations by reconstructing masked tokens, and fuse rich multi-modal knowledge in a common space. In addition, the cross-modal similarity matching module is proposed to optimize the intra-domain classification and decrease the inter-domain divergence. Extensive experiments are conducted on three public benchmarks CUHK-PEDES, ICFG-PEDES and RSTPReid to verify the effectiveness of our proposed framework. And results prove that our model outperforms state-of-the-art methods on all metrics.

Augmenting Multimodal Content Representation with Transformers for Misinformation Detection

Information sharing on social media has become a common practice for people around the world. Since it is difficult to check user-generated content on social media, huge amounts of rumors and misinformation are being spread with authentic information. On the one hand, most of the social platforms identify rumors through manual fact-checking, which is very inefficient. On the other hand, with an emerging form of misinformation that contains inconsistent image–text pairs, it would be beneficial if we could compare the meaning of multimodal content within the same post for detecting image–text inconsistency. In this paper, we propose a novel approach to misinformation detection by multimodal feature fusion with transformers and credibility assessment with self-attention-based Bi-RNN networks. Firstly, captions are derived from images using an image captioning module to obtain their semantic descriptions. These are compared with surrounding text by fine-tuning transformers for consistency check in semantics. Then, to further aggregate sentiment features into text representation, we fine-tune a separate transformer for text sentiment classification, where the output is concatenated to augment text embeddings. Finally, Multi-Cell Bi-GRUs with self-attention are used to train the credibility assessment model for misinformation detection. From the experimental results on tweets, the best performance with an accuracy of 0.904 and an F1-score of 0.921 can be obtained when applying feature fusion of augmented embeddings with sentiment classification results. This shows the potential of the innovative way of applying transformers in our proposed approach to misinformation detection. Further investigation is needed to validate the performance on various types of multimodal discrepancies.

Drug and Vaccine Extractive Text Summarization Insights Using Fine-tuned Transformers

Text representation is a key aspect in determining the success of various text summarizing techniques. Summarization using pre-trained transformer models has produced encouraging results. Yet the scope of applying these models in medical and drug discovery is not examined to a proper extent. To address this issue, this article aims to perform extractive summarization based on fine-tuned transformers pertaining to drug and medical domain. This research also aims to enhance sentence representation. Exploring the extractive text summarization aspects of medical and drug discovery is a challenging task as the datasets are limited. Hence, this research concentrates on the collection of abstracts collected from PubMed for various domains of medical and drug discovery such as drug, COVID, etc.with a total capacity of 1,370 abstracts. A detailed experimentation using BART (Bidirectional Autoregressive Transformer), T5(Text-to-Text Transfer Transformer), LexRank and TexRank for the analysis of the dataset is carried out in this research to perform extractive text summarization.

Going Underground: The Author’s Body and the Mining Workplace in the Writing about Miners in Modern China

Abstract This article highlights the author’s body and its physical experience in labor history, by a focus on their historical implication for the emergence of writings about miners in late-nineteenth- and early-twentieth century China. Miners and their workplace entered Chinese textual and pictorial representations at the turn of the twentieth century, as a result of the growing importance of the mining industry, the dissemination of new mining knowledge, and the formation of new social groups. Later in the 1920s, literary movements tied up with the revolutionary agenda under the influence of Marxism brought new conceptualizations of miners’ position in society and their relationship with intellectuals. When authors turned to write about miners in the late 1920s and early 1930s, they chose to legitimize their writing by descending into mine shafts to acquire the experience in the underground mining workplace and to incorporate the bodily experience into their writings. The mining lamp in texts and images from this period provides a prime example to illustrate how the author’s experience in the underground mine was embodied in the author’s and the miner’s corporeal visions and contributed a new way of seeing and representing miners. Drawing upon materials about miners of various genres, this article reveals the formulation of Chinese visual modernity within Chinese social interactions and brings the author’s body into labor history by calling attention to both its material presence in the workplace and its agentic power to inscribe workers.

GGAS2SN: Gated Graph and SmilesToSeq Network for Solubility Prediction.

Aqueous solubility is a critical physicochemical property of drug discovery. Solubility is a key issue in pharmaceutical development because it can limit a drug's absorption capacity. Accurate solubility prediction is crucial for pharmacological, environmental, and drug development studies. This research introduces a novel method for solubility prediction by combining gated graph neural networks (GGNNs) and graph attention neural networks (GATs) with Smiles2Seq encoding. Our methodology involves converting chemical compounds into graph structures with nodes representing atoms and edges indicating chemical bonds. These graphs are then processed by using a specialized graph neural network (GNN) architecture. Incorporating attention mechanisms into GNN allows for capturing subtle structural dependencies, fostering improved solubility predictions. Furthermore, we utilized the Smiles2Seq encoding technique to bridge the semantic gap between molecular structures and their textual representations. Smiles2Seq seamlessly converts chemical notations into numeric sequences, facilitating the efficient transfer of information into our model. We demonstrate the efficacy of our approach through comprehensive experiments on benchmark solubility data sets, showcasing superior predictive performance compared to traditional methods. Our model outperforms existing solubility prediction models and provides interpretable insights into the molecular features driving solubility behavior. This research signifies an important advancement in solubility prediction, offering potent tools for drug discovery, formulation development, and environmental assessments. The fusion of GGNN and Smiles2Seq encoding establishes a robust framework for accurately forecasting solubility across various chemical compounds, fostering innovation in various domains reliant on solubility data.

Exploring Recent NLP Advances for Tamil: Word Vectors and Hybrid Deep Learning Architectures

The advancements of deep learning methods and the availability of large corpora and data sets have led to an exponential increase in the performance of Natural Language Processing (NLP) methods resulting in successful NLP applications for various day-to-day tasks such as Language translation, Voice to text, Grammar checking, Sentiment analysis, etc. These advancements enabled the well-resourced languages to adapt themselves to the digital era while the gap for the low-resource languages widened. This research work explores the suitability of the recent advancements in NLP for Tamil, a low-resource language spoken mainly in South India, Sri Lanka, and Malaysia. From the literature survey, it has been found that there is a lack of comprehensive study on the effect of the recent advancements of NLP for the Tamil text. To fill this gap, this research work analysed the performance of deep learning based text representation and classification approaches namely word embedding, Convolutional Neural Network (CNN), and Recurrent Neural Network (RNN) for Tamil text classification tasks. Different dimensional pretrained word2Vec and FastText word vectors were built for Tamil and their effectiveness on Text classification was evaluated. The study found that the pre-trained 300- dimensional FastText word vector showed better performance than other pre-trained word vectors for Tamil text classification. Further, in this study, four simple hybrid CNN and Bi-GRU models were proposed for Tamil text classification and their performances were evaluated. The study found that hybrid CNN and Bi-GRU models perform better compared to the classical machine learning models, individual CNN and RNN models, and the Multilingual BERT model. These results confirm that the jointly learned embeddings with different deep learning architectures like CNN and RNN can achieve remarkable results for Tamil text classification, thus ensuring that the deep learning approaches can be successful for NLP on Tamil text.

Joint intent detection and slot filling with syntactic and semantic features using multichannel CNN-BiLSTM

Understanding spoken language is crucial for conversational agents, with intent detection and slot filling being the primary tasks in natural language understanding (NLU). Enhancing the NLU tasks can lead to an accurate and efficient virtual assistant thereby reducing the need for human intervention and expanding their applicability in other domains. Traditionally, these tasks have been addressed individually, but recent studies have highlighted their interconnection, suggesting better results when solved together. Recent advances in natural language processing have shown that pretrained word embeddings can enhance text representation and improve the generalization capabilities of models. However, the challenge of poor generalization in joint learning models for intent detection and slot filling remains due to limited annotated datasets. Additionally, traditional models face difficulties in capturing both the semantic and syntactic nuances of language, which are vital for accurate intent detection and slot filling. This study proposes a hybridized text representation method using a multichannel convolutional neural network with three embedding channels: non-contextual embeddings for semantic information, part-of-speech (POS) tag embeddings for syntactic features, and contextual embeddings for deeper contextual understanding. Specifically, we utilized word2vec for non-contextual embeddings, one-hot vectors for POS tags, and bidirectional encoder representations from transformers (BERT) for contextual embeddings. These embeddings are processed through a convolutional layer and a shared bidirectional long short-term memory (BiLSTM) network, followed by two softmax functions for intent detection and slot filling. Experiments on the air travel information system (ATIS) and SNIPS datasets demonstrated that our model significantly outperformed the baseline models, achieving an intent accuracy of 97.90% and slot filling F1-score of 98.86% on the ATIS dataset, and an intent accuracy of 98.88% and slot filling F1-score of 97.07% on the SNIPS dataset. These results highlight the effectiveness of our proposed approach in advancing dialogue systems, and paving the way for more accurate and efficient natural language understanding in real-world applications.

Social Event Classification Based on Multimodal Masked Transformer Network

The key to multimodal social event classification is to fully and accurately utilize the features of both image and text modalities. However, most existing methods have the following limitations: (1) they simply concatenate the image features and text features of the event, and (2) there is irrelevant contextual information between different modalities, which leads to mutual interference. Therefore, it is not enough to only consider the relationship between the modalities of multimodal data, but also the irrelevant contextual information (i.e., regions or words) between the modalities. To overcome these limitations, a novel social event classification method based on multimodal masked transformer network (MMTN) is proposed. A better representation of text and image is learned through an image-text encoding network. Then, the obtained image and text representations are input into the multimodal masked transformer network to fuse the multimodal information, and the relationship between the modalities of multimodal information is modeled by calculating the similarity between the multimodal information, masking the irrelevant context between the modalities. Extensive experiments on two benchmark datasets show that the proposed multimodal masked transformer network model achieves state-of-the-art performance.

Multi-Modal Contextualization of Trajectory Data for Advanced Analysis

SummaryRising amounts of generated geospatial data, either trajectory-like tracking data, raster-like imagery, or vector-like mappings as in OpenStreetMap (OSM), grow the need for multi-modal algorithmic analysis. Existing machine-learning-based algorithms contradictly mainly focus on image and textual input representations and cannot deal with other modes of geospatial data. Therefore, we propose a novel method to contextualize vector-like trajectory data with surrounding data to create easy-to-be-analyzed image-like representations. Our approach includes the proposition of a chase-cam-like scanline over space according to the trajectory’s speed and possibly smoothed orientation. Thereby, surrounding pixels in the vicinity of the trajectory points are accumulated along the scanline and are combined into a visual representation of the trajectory. To show the potential effects of our work, we predict traffic regulations for trajectory sections in the vehicle speed dataset based on our proposed trajectory-based sampling of orthophotos in the same region. This proposes a new way of using multi-modal data sources (trajectories and airborne imagery) to extract road metadata.

La caricatura política como signo visual durante el primer trimestre del gobierno del presidente Daniel Noboa.

This study analyzes the political cartoons of Xavier Bonilla (Bonil) and Vilma Vargas (Vilmatraca) in Ecuador during the beginning of Daniel Noboa's government (November 23, 2023 - February 25, 2024), focusing on five critical themes: Inauguration of Power, Popular Consultation, Drug Trafficking, VAT Increase, and Diplomatic Crisis with Russia. It explores the visual and textual representations in the intentionally selected works, analyzing drawing techniques, use of colors, facial expressions, and cultural references. This study shows how Bonil and Vilmatraca use signs and symbols to critique and comment on the political and social situation, demonstrating semiotics as an essential tool for understanding visual communication in politics. The study highlights the role of cartoons in provoking reflection on the Ecuadorian reality.

Leveraging Contrastive Language–Image Pre-Training and Bidirectional Cross-attention for Multimodal Keyword Spotting

In resource-limited keyword spotting scenarios, the scarcity of annotated corpora hinders deep learning’s ability to develop robust models for representing acoustic features. Recent studies focus on contrastive learning, using paired examples for self-supervision, indicating a growing research interest in keyword spotting. Typically, models are trained on segmented spoken words. However, the absence of distinct word boundaries in lengthy audio poses significant challenges. This study presents a novel approach that integrates Contrastive Language–Image Pre-Training with Cross-Attention for Self-Supervised Alignment in the field of Multimodal Keyword Spotting. The proposed method introduces a cross-modal process for matching word pairs, enhancing collaborative effectiveness between audio and text embeddings, achieving word-level congruence across different modalities. Employing a self-supervised learning approach, we use a restricted number of annotated audio–text pairs to discern semantic congruities and divergences, resulting in improved multimodal feature representation. Using this strategy, we input into subsequent keyword detection tasks, adopting a Bidirectional Cross-Attention block for detailed coordination across modalities, bridging the semantic gap between diverse audio and text representations, consequently improving keyword localization precision. Comprehensive experiments conducted on the Aishell-2 and Librispeech datasets demonstrate that the proposed method markedly surpasses existing techniques, especially regarding the comprehensive performance metric F1 for keyword spotting. It attains state-of-the-art outcomes for both pristine and noisy speech, showing respective enhancements of 5.2% and 5.4% on Aishell-2, as well as 5.4% and 5.6% on Librispeech.

AtCAF: Attention-based causality-aware fusion network for multimodal sentiment analysis

Multimodal sentiment analysis (MSA) involves interpreting sentiment using various sensory data modalities. Traditional MSA models often overlook causality between modalities, resulting in spurious correlations and ineffective cross-modal attention. To address these limitations, we propose the Attention-based Causality-Aware Fusion (AtCAF) network from a causal perspective. To capture a causality-aware representation of text, we introduce the Causality-Aware Text Debiasing Module (CATDM) utilizing the front-door adjustment. Furthermore, we employ the Counterfactual Cross-modal Attention (CCoAt) module integrate causal information in modal fusion, thereby enhancing the quality of aggregation by incorporating more causality-aware cues. AtCAF achieves state-of-the-art performance across three datasets, demonstrating significant improvements in both standard and Out-Of-Distribution (OOD) settings. Specifically, AtCAF outperforms existing models with a 1.5% improvement in ACC-2 on the CMU-MOSI dataset, a 0.95% increase in ACC-7 on the CMU-MOSEI dataset under normal conditions, and a 1.47% enhancement under OOD conditions. CATDM improves category cohesion in feature space, while CCoAt accurately classifies ambiguous samples through context filtering. Overall, AtCAF offers a robust solution for social media sentiment analysis, delivering reliable insights by effectively addressing data imbalance. The code is available at https://github.com/TheShy-Dream/AtCAF.

Towards fine-grained adaptive video captioning via Quality-Aware Recurrent Feedback Network

To achieve accurate video captioning, most existing solutions focus on employing auxiliary features to provide complementary generation clues, such as detected target regions, audio speech signals, and in-plane text. However, beyond introducing external information, the potential of using the generated captions has been overlooked. We argue that the intrinsic value lies in the caption text, which can faithfully reflect the video content. To this end, we propose to reuse the captions via a Quality-Aware Recurrent Feedback Network (QARFNet), a model that progressively exploits the supportive power between video features and the generated captions. Specifically, inspired by the backtracking AdaBoost algorithm, we propose to build a recurrent loop structure that recycles the obtained linguistic predictions to refine the input visual features. Referring to the greedy selective nature of the AdaBoost algorithm, we set a quality-aware gate to assess the necessity of forwarding the next loop, lessening the complexity. After the refinement loop, if the confidence score remains below the predefined threshold, we select the output with the highest confidence score to advance the subsequent transformer decoder. To facilitate feedback in each loop, a Multi-level Update Module is constructed before fusing linguistic predictions with video features. This involves extracting nouns and verbs from linguistic predictions to lighten relevant video tokens, achieving self-interactions across recurrent loops. By merging the coarse-grained and gradually refined linguistic prediction features, the video tokens are semantically closer to the desired textual representation. To alleviate the increased calculation caused by the loops, we employ a flat transformer encoder to decrease the complexity. The experimental results on several benchmark datasets confirm the effectiveness of our approach, with superior performance compared to the state-of-the-art.