Semantic Embedding Research Articles

Exploring a learning-to-rank approach to enhance the Retrieval Augmented Generation (RAG)-based electronic medical records search engines

BackgroundThis study addresses the challenge of enhancing Retrieval Augmented Generation (RAG) search engines for electronic medical records (EMR) by learning users' distinct search semantics. The specific aim is to develop a learning-to-rank system that improves the accuracy and relevance of search results to support RAG-based search engines. MethodsGiven a prompt or search query, the system first asks the user to label a few randomly selected documents, which contain some keywords, as relevant to the prompt or not. The system then identifies relevant sentences and adjusts word similarities by updating a medical semantic embedding. New documents are ranked by the number of relevant sentences identified by the weighted embedding. Only the top-ranked documents and sentences are provided to a Large-Language-Model (LLM) to generate answers for further review. FindingsTo evaluate our approach, four medical researchers labeled documents based on their relevance to specific diseases. We measured the information retrieval performance of our approach and two baseline methods. Results show that our approach achieved at least a 0.60 Precision-at-10 (P @ 10) score with only ten positive labels, outperforming the baseline methods. In our pilot study, we demonstrate that the learned semantic preference can transfer to the analysis of unseen datasets, boosting the accuracy of an RAG model in extracting and explaining cancer progression diagnoses from 0.14 to 0.50. InterpretationThis study demonstrates that a customized learning-to-rank method can enhance state-of-the-art natural language models, such as LLMs, by quickly adapting to users' semantics. This approach supports EMR document retrieval and helps RAG models generate clinically meaningful answers to specific questions, underscoring the potential of user-tailored learning-to-rank methods in clinical practice.

Memory-Based Cross-Modal Semantic Alignment Network for Radiology Report Generation.

Generating radiology reports automatically reduces the workload of radiologists and helps the diagnoses of specific diseases. Many existing methods take this task as modality transfer process. However, since the key information related to disease accounts for a small proportion in both image and report, it is hard for the model to learn the latent relation between the radiology image and its report, thus failing to generate fluent and accurate radiology reports. To tackle this problem, we propose a memory-based cross-modal semantic alignment model (MCSAM) following an encoder-decoder paradigm. MCSAM includes a well initialized long-term clinical memory bank to learn disease-related representations as well as prior knowledge for different modalities to retrieve and use the retrieved memory to perform feature consolidation. To ensure the semantic consistency of the retrieved cross modal prior knowledge, a cross-modal semantic alignment module (SAM) is proposed. SAM is also able to generate semantic visual feature embeddings which can be added to the decoder and benefits report generation. More importantly, to memorize the state and additional information while generating reports with the decoder, we use learnable memory tokens which can be seen as prompts. Extensive experiments demonstrate the promising performance of our proposed method which generates state-of-the-art performance on the MIMIC-CXR dataset.

SS-CRE: A Continual Relation Extraction Method Through SimCSE-BERT and Static Relation Prototypes

Continual relation extraction aims to learn new relations from a continuous stream of data while avoiding forgetting old relations. Existing methods typically use the BERT encoder to obtain semantic embeddings, ignoring the fact that the vector representations suffer from anisotropy and uneven distribution. Furthermore, the relation prototypes are usually computed by memory samples directly, resulting in the model being overly sensitive to memory samples. To solve these problems, we propose a new continual relation extraction method. Firstly, we modified the basic structure of the sample encoder to generate uniformly distributed semantic embeddings using the supervised SimCSE-BERT to obtain richer sample information. Secondly, we introduced static relation prototypes and dynamically adjust their proportion with dynamic relation prototypes to adapt to the feature space. Lastly, through experimental analysis on the widely used FewRel and TACRED datasets, the results demonstrate that the proposed method effectively enhances semantic embeddings and relation prototypes, resulting in a further alleviation of catastrophic forgetting in the model. The code will be soon released at https://github.com/SuyueW/SS-CRE.

TA-Detector: A GNN-based Anomaly Detector via Trust Relationship

With the rise of mobile Internet and AI, social media integrating short messages, images, and videos has developed rapidly. As a guarantee for the stable operation of social media, information security, especially graph anomaly detection (GAD), has become a hot issue inspired by the extensive attention of researchers. Most GAD methods are mainly limited to enhancing the homophily or considering homophily and heterophilic connections. Nevertheless, due to the deceptive nature of homophily connections among anomalies, the discriminative information of the anomalies can be eliminated. To alleviate the issue, we explore a novel method TA-Detector in GAD by introducing the concept of trust into the classification of connections. In particular, the proposed approach adopts a designed trust classier to distinguish trust and distrust connections with the supervision of labeled nodes. Then, we capture the latent factors related to GAD by graph neural networks (GNN), which integrate node interaction type information and node representation. Finally, to identify anomalies in the graph, we use the residual network mechanism to extract the deep semantic embedding information related to GAD. Experimental results on two real benchmark datasets verify that our proposed approach boosts the overall GAD performance in comparison to benchmark baselines.

Online Cross-modal Hashing With Dynamic Prototype

Online cross-modal hashing has received increasing attention due to its efficiency and effectiveness in handling cross-modal streaming data retrieval. Despite the promising performance, these methods mainly focus on the supervised learning paradigm, demanding expensive and laborious work to obtain clean annotated data. Existing unsupervised online hashing methods mostly struggle to construct instructive semantic correlations among data chunks, resulting in the forgetting of accumulated data distribution. To this end, we propose a Dynamic Prototype-based Online Cross-modal Hashing method, called DPOCH. Based on the pre-learned reliable common representations, DPOCH generates prototypes incrementally as sketches of accumulated data and updates them dynamically for adapting streaming data. Thereafter, the prototype-based semantic embedding and similarity graphs are designed to promote stability and generalization of the hashing process, thereby obtaining globally adaptive hash codes and hash functions. Experimental results on benchmarked datasets demonstrate that the proposed DPOCH outperforms state-of-the-art unsupervised online cross-modal hashing methods.

ROSE: Multi-level super-resolution-oriented semantic embedding for 3D microvasculature segmentation from low-resolution images

Current state-of-the-art segmentation methods often require high-resolution input to attain the high performance, which pushes the limit of data acquisition and brings large computation budgets. Instead, we present an end-to-end deep learning-based method, ROSE, for robust and precise segmentation of high-quality 3D super-resolution (SR) microvasculatures from low-resolution (LR) images as input, which can transform data from the LR imaging domain to the SR semantic domain (cross different modalities and scales). More specifically, a multi-tasking two-stream deep learning framework is proposed to learn the high-fidelity microvasculature SR image and semantic hybrid features simultaneously. During the proposed joint learning process, the high-resolution features of microvasculatures are further enhanced by the learned fine-grained structural/textural features from the microvasculature SR stream with a multi-level embedding scheme through the oriented feature aggregation at different fusion stages. In the constructed joint multi-level hybrid embedding spaces, the instance semantic embedding and the SR imaging embedding can be connected and integrated synergistically. We have conducted extensive experiments using public and real patient micro-cerebrovascular image datasets and compare our framework with traditional 3D vessel segmentation methods and the other state-of-the-art in deep learning. This robust and precise microvascular visualization in different brain regions by our method demonstrates its potential impact in magnetic resonance (MR) angiography and venography for the diagnosis of microvascular disease.

Semantic aware-based instruction embedding for binary code similarity detection.

Binary code similarity detection plays a crucial role in various applications within binary security, including vulnerability detection, malicious software analysis, etc. However, existing methods suffer from limited differentiation in binary embedding representations across different compilation environments, lacking dynamic high-level semantics. Moreover, current approaches often neglect multi-level semantic feature extraction, thereby failing to acquire precise semantic information about the binary code. To address these limitations, this paper introduces a novel detection solution called BinBcla. This method employs an enhanced pre-training model to generate instruction embeddings with dynamic semantics for binary functions. Subsequently, multi-feature fusion technique is utilized to extract local semantic information and long-distance global features from the code, respectively, employing self-attention to comprehend the structure information of the code. Finally, an improved cosine similarity method is employed to learn relationships among all elements of the distance vectors, thereby enhancing the model's robustness to new sample functions. Experiments are conducted across different architectures, compilers, and optimization levels. The results indicate that BinBcla achieves higher accuracy, precision and F1 score compared to existing methods.

Neural models for semantic analysis of handwritten document images

Semantic analysis of handwritten document images offers a wide range of practical application scenarios. A sequential combination of handwritten text recognition (HTR) and a task-specific natural language processing system offers an intuitive solution in this domain. However, this HTR-based approach suffers from the problem of error propagation. An HTR-free model, which avoids explicit text recognition and solves the task end-to-end, tackles this problem, but often produces poor results. A possible reason for this is that it does not incorporate largely pre-trained semantic word embeddings, which turn out to be one of the most powerful advantages in the textual domain. In this work, we propose an HTR-based and an HTR-free model and compare them on a variety of segmentation-based handwritten document image benchmarks including semantic word spotting, named entity recognition, and question answering. Furthermore, we propose a cross-modal knowledge distillation approach to integrate semantic knowledge from textually pre-trained word embeddings into HTR-free models. In a series of experiments, we investigate optimization strategies for robust semantic word image representation. We show that the incorporation of semantic knowledge is beneficial for HTR-free approaches in achieving state-of-the-art results on a variety of benchmarks.

Swift Detection of XSS Attacks: Enhancing XSS Attack Detection by Leveraging Hybrid Semantic Embeddings and AI Techniques

AbstractCross-Site Scripting (XSS) attacks continue to be a significant threat to web application security, necessitating robust detection mechanisms to safeguard user data and ensure system integrity. In this study, we present a novel approach for detecting XSS attacks that harnesses the combined capabilities of the Universal Sentence Encoder (USE) and Word2Vec embeddings as a feature extractor, aiming to enhance the performance of machine learning and deep learning techniques. By leveraging the semantic understanding of sentences offered by USE and the word-level representations from Word2Vec, we obtain a comprehensive feature representation for XSS attack payloads. Our proposed approach aims to capture both fine-grained word meanings and broader sentence contexts, leading to enhanced feature extraction and improved model performance. We conducted extensive experiments utilizing machine learning and deep learning architectures to evaluate the effectiveness of our approach. The obtained results demonstrate that our combined embeddings approach outperforms traditional methods, achieving superior accuracy, precision, recall, ROC, and F1-score in detecting XSS attacks. This study not only advances XSS attack detection but also highlights the potential of state-of-the-art natural language processing techniques in web security applications. Our findings offer valuable insights for the development of more robust and effective security measures against XSS attacks.

A Survey of Visual Transformers.

Transformer, an attention-based encoder-decoder model, has already revolutionized the field of natural language processing (NLP). Inspired by such significant achievements, some pioneering works have recently been done on employing Transformer-liked architectures in the computer vision (CV) field, which have demonstrated their effectiveness on three fundamental CV tasks (classification, detection, and segmentation) as well as multiple sensory data stream (images, point clouds, and vision-language data). Because of their competitive modeling capabilities, the visual Transformers have achieved impressive performance improvements over multiple benchmarks as compared with modern convolution neural networks (CNNs). In this survey, we have reviewed over 100 of different visual Transformers comprehensively according to three fundamental CV tasks and different data stream types, where taxonomy is proposed to organize the representative methods according to their motivations, structures, and application scenarios. Because of their differences on training settings and dedicated vision tasks, we have also evaluated and compared all these existing visual Transformers under different configurations. Furthermore, we have revealed a series of essential but unexploited aspects that may empower such visual Transformers to stand out from numerous architectures, e.g., slack high-level semantic embeddings to bridge the gap between the visual Transformers and the sequential ones. Finally, two promising research directions are suggested for future investment. We will continue to update the latest articles and their released source codes at https://github.com/liuyang-ict/awesome-visual-transformers.

ChineseEEG: A Chinese Linguistic Corpora EEG Dataset for Semantic Alignment and Neural Decoding

An Electroencephalography (EEG) dataset utilizing rich text stimuli can advance the understanding of how the brain encodes semantic information and contribute to semantic decoding in brain-computer interface (BCI). Addressing the scarcity of EEG datasets featuring Chinese linguistic stimuli, we present the ChineseEEG dataset, a high-density EEG dataset complemented by simultaneous eye-tracking recordings. This dataset was compiled while 10 participants silently read approximately 13 hours of Chinese text from two well-known novels. This dataset provides long-duration EEG recordings, along with pre-processed EEG sensor-level data and semantic embeddings of reading materials extracted by a pre-trained natural language processing (NLP) model. As a pilot EEG dataset derived from natural Chinese linguistic stimuli, ChineseEEG can significantly support research across neuroscience, NLP, and linguistics. It establishes a benchmark dataset for Chinese semantic decoding, aids in the development of BCIs, and facilitates the exploration of alignment between large language models and human cognitive processes. It can also aid research into the brain’s mechanisms of language processing within the context of the Chinese natural language.

Global Semantic-Sense Aggregation Network for Salient Object Detection in Remote Sensing Images.

Salient object detection (SOD) aims to accurately identify significant geographical objects in remote sensing images (RSI), providing reliable support and guidance for extensive geographical information analyses and decisions. However, SOD in RSI faces numerous challenges, including shadow interference, inter-class feature confusion, as well as unclear target edge contours. Therefore, we designed an effective Global Semantic-aware Aggregation Network (GSANet) to aggregate salient information in RSI. GSANet computes the information entropy of different regions, prioritizing areas with high information entropy as potential target regions, thereby achieving precise localization and semantic understanding of salient objects in remote sensing imagery. Specifically, we proposed a Semantic Detail Embedding Module (SDEM), which explores the potential connections among multi-level features, adaptively fusing shallow texture details with deep semantic features, efficiently aggregating the information entropy of salient regions, enhancing information content of salient targets. Additionally, we proposed a Semantic Perception Fusion Module (SPFM) to analyze map relationships between contextual information and local details, enhancing the perceptual capability for salient objects while suppressing irrelevant information entropy, thereby addressing the semantic dilution issue of salient objects during the up-sampling process. The experimental results on two publicly available datasets, ORSSD and EORSSD, demonstrated the outstanding performance of our method. The method achieved 93.91% Sα, 98.36% Eξ, and 89.37% Fβ on the EORSSD dataset.

Big data analysis for Covid-19 in hospital information systems.

The COVID-19 pandemic has triggered a global public health crisis, affecting hundreds of countries. With the increasing number of infected cases, developing automated COVID-19 identification tools based on CT images can effectively assist clinical diagnosis and reduce the tedious workload of image interpretation. To expand the dataset for machine learning methods, it is necessary to aggregate cases from different medical systems to learn robust and generalizable models. This paper proposes a novel deep learning joint framework that can effectively handle heterogeneous datasets with distribution discrepancies for accurate COVID-19 identification. We address the cross-site domain shift by redesigning the COVID-Net's network architecture and learning strategy, and independent feature normalization in latent space to improve prediction accuracy and learning efficiency. Additionally, we propose using a contrastive training objective to enhance the domain invariance of semantic embeddings and boost classification performance on each dataset. We develop and evaluate our method with two large-scale public COVID-19 diagnosis datasets containing CT images. Extensive experiments show that our method consistently improves the performance both datasets, outperforming the original COVID-Net trained on each dataset by 13.27% and 15.15% in AUC respectively, also exceeding existing state-of-the-art multi-site learning methods.

Improving plant miRNA-target prediction with self-supervised k-mer embedding and spectral graph convolutional neural network.

Deciphering the targets of microRNAs (miRNAs) in plants is crucial for comprehending their function and the variation in phenotype that they cause. As the highly cell-specific nature of miRNA regulation, recent computational approaches usually utilize expression data to identify the most physiologically relevant targets. Although these methods are effective, they typically require a large sample size and high-depth sequencing to detect potential miRNA-target pairs, thereby limiting their applicability in improving plant breeding. In this study, we propose a novel miRNA-target prediction framework named kmerPMTF (k-mer-based prediction framework for plant miRNA-target). Our framework effectively extracts the latent semantic embeddings of sequences by utilizing k-mer splitting and a deep self-supervised neural network. We construct multiple similarity networks based on k-mer embeddings and employ graph convolutional networks to derive deep representations of miRNAs and targets and calculate the probabilities of potential associations. We evaluated the performance of kmerPMTF on four typical plant datasets: Arabidopsis thaliana, Oryza sativa, Solanum lycopersicum, and Prunus persica. The results demonstrate its ability to achieve AUPRC values of 84.9%, 91.0%, 80.1%, and 82.1% in 5-fold cross-validation, respectively. Compared with several state-of-the-art existing methods, our framework achieves better performance on threshold-independent evaluation metrics. Overall, our study provides an efficient and simplified methodology for identifying plant miRNA-target associations, which will contribute to a deeper comprehension of miRNA regulatory mechanisms in plants.

Fusing Domain-Specific Content from Large Language Models into Knowledge Graphs for Enhanced Zero Shot Object State Classification

Domain-specific knowledge can significantly contribute to addressing a wide variety of vision tasks. However, the generation of such knowledge entails considerable human labor and time costs. This study investigates the potential of Large Language Models (LLMs) in generating and providing domain-specific information through semantic embeddings. To achieve this, an LLM is integrated into a pipeline that utilizes Knowledge Graphs and pre-trained semantic vectors in the context of the Vision-based Zero-shot Object State Classification task. We thoroughly examine the behavior of the LLM through an extensive ablation study. Our findings reveal that the integration of LLM-based embeddings, in combination with general-purpose pre-trained embeddings, leads to substantial performance improvements. Drawing insights from this ablation study, we conduct a comparative analysis against competing models, thereby highlighting the state-of-the-art performance achieved by the proposed approach.

RQ-OSPTrans: A Semantic Classification Method Based on Transformer That Combines Overall Semantic Perception and “Repeated Questioning” Learning Mechanism

The pre-trained language model based on Transformers possesses exceptional general text-understanding capabilities, empowering it to adeptly manage a variety of tasks. However, the topic classification ability of the pre-trained language model will be seriously affected in the face of long colloquial texts, expressions with similar semantics but completely different expressions, and text errors caused by partial speech recognition. We propose a long-text topic classification method called RQ-OSPTrans to effectively address these challenges. To this end, two parallel learning modules are proposed to learn long texts, namely, the repeat question module and the overall semantic perception module. The overall semantic perception module will conduct average pooling on the semantic embeddings produced by BERT, in addition to multi-layer perceptron learning. The repeat question module will learn the text-embedding matrix, extracting detailed clues for classification based on words as fundamental elements. Comprehensive experiments demonstrate that RQ-OSPTrans can achieve a generalization performance of 98.5% on the Chinese dataset THUCNews. Moreover, RQ-OSPTrans can achieve state-of-the-art performance on the arXiv-10 dataset (84.4%) and has a comparable performance with other state-of-the-art pre-trained models on the AG’s News dataset. Finally, the results indicate that our method exhibits a superior performance compared with the baseline methods on small-scale domain-specific datasets by validating RQ-OSPTrans on a specific task scenario by using our custom-built dataset CCIPC.

TFSemantic: A Time–Frequency Semantic GAN Framework for Imbalanced Classification Using Radio Signals

Recently, wireless sensing techniques have been widely used for Internet of Things (IoT) applications. Unlike traditional device-based sensing, wireless sensing is contactless, pervasive, low cost, and non-invasive, making it highly suitable for relevant IoT applications. However, most existing methods are highly dependent on high-quality datasets, and the minority class will not achieve a satisfactory performance when suffering from a class imbalance problem. In this article, we propose a time–frequency semantic generative adversarial network framework (i.e., TFSemantic) to address the imbalanced classification problem in human activity recognition using radio frequency (RF) signals. Specifically, the TFSemantic framework can learn semantic features from the minority classes and then generate high-quality signals to restore data balance. It includes a data pre-processing module, a semantic extraction module, a semantic distribution module, and a data augmenter module. In the data pre-processing module, we process four different RF datasets (i.e., WiFi, RFID, UWB, and mmWave). We also develop Fourier semantic feature convolution and attention semantic feature embedding methods for the semantic extraction module. A discrete wavelet transform is utilized for reconstructed RF samples in the semantic distribution module. In data augmenter module, we design an associated loss function to achieve effective adversarial training. Finally, we validate the effectiveness of the proposed TFSemantic framework using different RF datasets, which outperforms several state-of-the-art methods.

GA-GGD: Improving semantic discriminability in graph contrastive learning via Generative Adversarial Network

Graph contrastive learning has garnered considerable research interest due to its ability to effectively embed graph data without manual labels. Among them, methods based on Deep Graph Infomax (DGI) have been widely studied and favored in the industry because of their fast training speed, applicability to large-scale data. DGI-based methods usually obtain a noise graph through node shuffling. The proxy task of these methods encourages the encoder to distinguish whether the nodes come from the original graph or the noise graph, thereby maximizing the mutual information between the node representation and the graph it belongs to, while also maximizing the Jenson–Shannon divergence between the nodes of original graph and noise graph. However, we argue that these approaches only enable the encoder to differentiate between semantically meaningful graphs and noise graphs, but not to effectively identify different semantic graphs. This leads to the inability of the encoder to effectively embed information between different semantics, significantly reducing the robustness and affecting the performance of downstream tasks. In addition, this training mode makes the model more sensitive to attacks. To improve their semantic discriminability, we take advantage of the natural ability of generative adversarial networks to generate semantic data, proposing a method called Generative Adversarial Graph Group Discrimination (GA-GGD). Specifically, it consists of a graph group discriminator and a semantic attack generator. The discriminator aims to encode the graph and identify whether nodes originate from the original graph. The goal of the generator is to use random features and graph structure to find vulnerabilities of discriminator and generate node representations with similar but wrong semantic to confuse the discriminator. GA-GGD can improve the model’s semantic information embedding without significantly increasing computational overhead and memory occupancy. We test the effectiveness of the proposed model on commonly used data sets and large-scale datasets, as well as in various downstream tasks such as classification, clustering, and adversarial attacks defence. A wealth of experimental results confirm the efficacy of the proposed model.

Take a prior from other tasks for severe blur removal

Recovering clear structures from severely blurry inputs is a huge challenge due to the detail loss and ambiguous semantics. Although segmentation maps can help deblur facial images, their effectiveness is limited in complex natural scenes because they ignore the detailed structures necessary for deblurring. Furthermore, direct segmentation of blurry images may introduce error propagation. To alleviate the semantic confusion and avoid error propagation, we propose utilizing high-level vision tasks, such as classification, to learn a comprehensive prior for severe blur removal. We propose a feature learning strategy based on knowledge distillation, which aims to learn the priors with global contexts and sharp local structures. To integrate the priors effectively, we propose a semantic prior embedding layer with multi-level aggregation and semantic attention. We validate our method on natural image deblurring benchmarks by introducing the priors to various models, including UNet and mainstream deblurring baselines, to demonstrate its effectiveness and generalization ability. The results show that our approach outperforms existing methods on severe blur removal with our plug-and-play semantic priors.

Zero-shot discrete hashing with adaptive class correlation for cross-modal retrieval

Zero-shot retrieval aims to transfer knowledge from seen classes to unseen classes by embedding semantic information on class attributes, thus solving the unseen class retrieval problem. However, existing works have focused mainly on unimodal zero-shot retrieval tasks. In this paper, we introduce an efficient method, termed zero-shot discrete hashing with adaptive class correlation (ZSDH-ACC), to speed up cross-modal retrieval. Specifically, this proposed method combines label information with class attribute information to construct a semantic enhancement embedding, in which the problem of class attribute correspondence of multilabel instances can be solved. Furthermore, we learn semantic enhancement embedding to merge more semantic information for feature representation, and its goal is to learn more discriminative hash codes and hash functions. In addition, our proposed method adaptively learns the correlation between class attributes and then embeds more class attribute information into hash codes. Finally, pairwise similarity is used to constrain the learning of hash codes, and thus more discriminative hash codes can be generated. Extensive experimental results on four benchmark multimodal datasets demonstrate that the proposed ZSDH-ACC method can achieve encouraging performance in cross-modal retrieval tasks. The source code of this paper can be obtained from https://github.com/szq0816/ZSDH_ACC.