Semantic Understanding Research Articles

Preserving and combining knowledge in robotic lifelong reinforcement learning

Abstract Humans can continually accumulate knowledge and develop increasingly complex behaviours and skills throughout their lives, which is a capability known as ‘lifelong learning’. Although this lifelong learning capability is considered an essential mechanism that makes up general intelligence, recent advancements in artificial intelligence predominantly excel in narrow, specialized domains and generally lack this lifelong learning capability. Here we introduce a robotic lifelong reinforcement learning framework that addresses this gap by developing a knowledge space inspired by the Bayesian non-parametric domain. In addition, we enhance the agent’s semantic understanding of tasks by integrating language embeddings into the framework. Our proposed embodied agent can consistently accumulate knowledge from a continuous stream of one-time feeding tasks. Furthermore, our agent can tackle challenging real-world long-horizon tasks by combining and reapplying its acquired knowledge from the original tasks stream. The proposed framework advances our understanding of the robotic lifelong learning process and may inspire the development of more broadly applicable intelligence.

The Scalable Detection and Resolution of Data Clumps Using a Modular Pipeline with ChatGPT

This paper explores a modular pipeline architecture that integrates ChatGPT, a Large Language Model (LLM), to automate the detection and refactoring of data clumps—a prevalent type of code smell that complicates software maintainability. Data clumps refer to clusters of code that are often repeated and should ideally be refactored to improve code quality. The pipeline leverages ChatGPT’s capabilities to understand context and generate structured outputs, making it suitable for addressing complex software refactoring tasks. Through systematic experimentation, our study not only addresses the research questions outlined but also demonstrates that the pipeline can accurately identify data clumps, particularly excelling in cases that require semantic understanding—where localized clumps are embedded within larger codebases. While the solution significantly enhances the refactoring workflow, facilitating the management of distributed clumps across multiple files, it also presents challenges such as occasional compiler errors and high computational costs. Feedback from developers underscores the usefulness of LLMs in software development but also highlights the essential role of human oversight to correct inaccuracies. These findings demonstrate the pipeline’s potential to enhance software maintainability, offering a scalable and efficient solution for addressing code smells in real-world projects, and contributing to the broader goal of enhancing software maintainability in large-scale projects.

Evaluating the effectiveness of machine learning methods for keyword coverage using semantic data analysis

This article presents a comprehensive comparative analysis of two advanced hybrid machine learning approaches for keyword extraction: bidirectional encoder representations from transformers (BERT) combined with autoencoder (AE) and term frequency-inverse document frequency (TF-IDF) combined with autoencoder. The research targets the task of semantic analysis in text data to evaluate the effectiveness of these methods in ensuring adequate keyword coverage across diverse text corpora. The study delves into the architecture and operational principles of each method, with a particular focus on the integration with autoencoders to enhance the semantic integrity and relevance of the extracted keywords. The experimental section provides a detailed performance analysis of both methods on various text datasets, highlighting how the structure and semantic richness of the source data influence the outcomes. The evaluation methodology includes precision, recall, and F1-score metrics. The paper discusses the advantages and disadvantages of each approach and their suitability for specific keyword extraction tasks. The findings offer valuable insights for the scientific community, aiding in the selection of the most appropriate text processing method for applications requiring deep semantic understanding and high accuracy in information extraction.

ProtoSAM-3D: Interactive semantic segmentation in volumetric medical imaging via a Segment Anything Model and mask-level prototypes.

Semantic segmentation of volumetric medical images is essential for accurate delineation of anatomic structures and pathology, enabling quantitative analysis in precision medicine applications. While volumetric segmentation has been extensively studied, most existing methods require full supervision and struggle to generalize to new classes at inference time, particularly for irregular, ill-defined targets such as tumors, where fine-grained, high-salience segmentation is required. Consequently, conventional semantic segmentation methods cannot easily offer zero/few-shot generalization to segment objects of interest beyond their closed training set. Foundation models, such as the Segment Anything Model (SAM), have demonstrated promising zero-shot generalization for interactive instance segmentation based on user prompts. However, these models sacrifice semantic knowledge for generalization capabilities that largely rely on collaborative user prompting to inject semantics. For volumetric medical image analysis, a unified approach that combines the semantic understanding of conventional segmentation methods with the flexible, prompt-driven capabilities of SAM is essential for comprehensive anatomical delineation On the one hand, it is natural to exploit anatomic knowledge to enable semantic segmentation without any user interaction. On the other hand, SAM-like approaches to segment unknown classes via prompting provide the needed flexibility to segment structures beyond the closed training set, enabling quantitative analysis. To address these needs in a unified framework, we introduce ProtoSAM-3D, which extends SAMs to semantic segmentation of volumetric data via a novel mask-level prototype prediction approach while retaining the flexibility of SAM Our model utilizes an innovative spatially-aware Transformer to fuse instance-specific intermediate representations from the SAM encoder and decoder, obtaining a comprehensive feature embedding for each mask. These embeddings are then classified by computing similarity with learned prototypes. By predicting prototypes instead of classes directly, ProtoSAM-3D gains the flexibility to rapidly adapt to new classes with minimal retraining. Furthermore, we introduce an auto-prompting method to enable semantic segmentation of known classes without user interaction. We demonstrate state-of-the-art zero/few-shot performance on multi-organ segmentation in CT and MRI. Experimentally, ProtoSAM-3D achieves competitive performance compared to fully supervised methods. Our work represents a step towards interactive semantic segmentation models with SAM for volumetric medical image processing.

Word embedding factor based multi-head attention

The natural language processing (NLP) field has made significant progress using deep learning models based on multi-head attention mechanisms, such as Transformer and BERT. However, there are two major limitations to this approach. First, the number of heads is often manually set based on empirical experience, and second, it is not clear enough in semantic understanding and interpretation. In this study, we propose a novel attention mechanism called Factor Analysis-based Multi-head (FAM) Attention, which combines the theory of explorative factor analysis and word embedding. The experimental results demonstrate that FAM Attention achieves better performance and requires fewer parameters compared to traditional methods while also having better semantic understanding ability and interpretability at the token level. This also has significant implications for current Large Language Models (LLMs), particularly in terms of effectively reducing parameter counts and enhancing performance.

High-Precision Image Editing via Dual Attention Control in Diffusion Models Without Fine-Tuning

Existing diffusion models outperform generative models like Generative Adversarial Networks in image synthesis and editing. However, they struggle with high-precision image editing while preserving image details and the accuracy of editing instructions. To address these challenges, we propose a dual attention control method to achieve high-precision image editing. Our approach includes two key attention control modules: (1) cross-attention control module, which combines the cross-attention maps of the original and edited images through weighted parameters, ensures that the synthesized edited image retains the structure of the input image. (2) Self-attention control module, which varies based on the editing task, applied at “coarse” and “fine” layers, since the coarse layers help maintain input image details and the fine layers are better suited for style transformations. Experimental evaluations have demonstrated that our approach achieves excellent results in detail preservation, content consistency, visual realism, and semantic understanding, making it especially suitable for tasks requiring high-precision editing. Specifically, compared to the editing outcomes under no control conditions, the introduction of dual visual attention control has led to an increase of 6.19% in CLIP scores, a reduction of 29.3% in LPIPS, and a decrease of 24.7% in FID. These significant improvements not only validate the effectiveness of the dual attention control but also attest to the method’s substantial flexibility and adaptability across different scenarios. Notably, our approach is a zero-shot solution, requiring no user optimization or fine-tuning, facilitating real-world applications.

Design of an intelligent English learning platform combining biomechanical analysis with biological data analysis and text semantic matching

Intelligent classrooms have demonstrated significant promise in enhancing learning efficiency as a result of the quick development of big data and artificial intelligence technologies. This study proposes a text semantic matching model (OM) that combines deep learning and K-means clustering algorithm, aiming to optimize vocabulary. Importantly, it delves into the biomechanical aspects of learning by considering how physical and physiological processes interact with language acquisition. By mimicking the learning mechanism of biological neural networks and further exploring the biomechanical correlates of neural activity during learning, such as the muscle tensions and postural changes associated with cognitive efforts, this model simulates how the brain processes and stores language information. These biomechanical factors can have an impact on concentration and fatigue levels, which in turn affect semantic understanding and memory performance during the learning process. The experimental results indicate that this method not only improves teaching effectiveness, but also provides a solid foundation for future research on intelligent language learning environments, taking into account the biomechanical underpinnings of learning.

A DeBERTa-Based Semantic Conversion Model for Spatiotemporal Questions in Natural Language

To address current issues in natural language spatiotemporal queries, including insufficient question semantic understanding, incomplete semantic information extraction, and inaccurate intent recognition, this paper proposes NL2Cypher, a DeBERTa (Decoding-enhanced BERT with disentangled attention)-based natural language spatiotemporal question semantic conversion model. The model first performs semantic encoding on natural language spatiotemporal questions, extracts pre-trained features based on the DeBERTa model, inputs feature vector sequences into BiGRU (Bidirectional Gated Recurrent Unit) to learn text features, and finally obtains globally optimal label sequences through a CRF (Conditional Random Field) layer. Then, based on the encoding results, it performs classification and semantic parsing of spatiotemporal questions to achieve question intent recognition and conversion to Cypher query language. The experimental results show that the proposed DeBERTa-based conversion model NL2Cypher can accurately achieve semantic information extraction and intent understanding in both simple and compound queries when using Chinese corpus, reaching an F1 score of 92.69%, with significant accuracy improvement compared to other models. The conversion accuracy from spatiotemporal questions to query language reaches 88% on the training set and 92% on the test set. The proposed model can quickly and accurately query spatiotemporal data using natural language questions. The research results provide new tools and perspectives for subsequent knowledge graph construction and intelligent question answering, effectively promoting the development of geographic information towards intelligent services.

Enhancing zero-shot stance detection via multi-task fine-tuning with debate data and knowledge augmentation

In the real world, stance detection tasks often involve assessing the stance or attitude of a given text toward new, unseen targets, a task known as zero-shot stance detection. However, zero-shot stance detection often suffers from issues such as sparse data annotation and inherent task complexity, which can lead to lower performance. To address these challenges, we propose combining fine-tuning of Large Language Models (LLMs) with knowledge augmentation for zero-shot stance detection. Specifically, we leverage stance detection and related tasks from debate corpora to perform multi-task fine-tuning of LLMs. This approach aims to learn and transfer the capability of zero-shot stance detection and reasoning analysis from relevant data. Additionally, we enhance the model’s semantic understanding of the given text and targets by retrieving relevant knowledge from external knowledge bases as context, alleviating the lack of relevant contextual knowledge. Compared to ChatGPT, our model achieves a significant improvement in the average F1 score, with an increase of 15.74% on the SemEval 2016 Task 6 A and 3.55% on the P-Stance dataset. Our model outperforms current state-of-the-art models on these two datasets, demonstrating the superiority of multi-task fine-tuning with debate data and knowledge augmentation.

Improved Grain Boundary Reconstruction Method Based on Channel Attention Mechanism.

The grain size of metal materials has a significant impact on their macroscopic properties. However, original metallographic images often suffer from issues such as substantial noise, missing grain boundaries, low contrast, and blurred edges. These challenges hinder the accurate extraction of complete grain boundaries, limiting the precision of grain size measurement and material performance prediction. Therefore, effectively reconstructing incomplete grain boundaries is particularly crucial. This paper proposes a grain boundary reconstruction and grain size measurement method based on an improved channel attention mechanism. A generative adversarial network (GAN) serves as the backbone, with a custom-designed channel attention module embedded in the generator. Combined with a global context attention mechanism, the method captures the global contextual information of the image, enhancing the network's semantic understanding and reconstruction accuracy for regions with missing grain boundaries. During the image reconstruction process, the method effectively leverages long-range feature correlations within the image, significantly improving network performance. To address the Mode Collapse observed during experiments, the loss function is optimized using Focal Loss, balancing the ratio of positive and negative samples and improving network robustness. Compared with other attention modules, the improved channel attention module significantly enhances the performance of the generative network. Experimental results demonstrate that the generative network based on this module outperforms comparable modules in terms of MIoU (86.25%), Accuracy (95.06%), and Precision (86.54%). The grain boundary reconstruction method based on the improved channel attention mechanism not only effectively improves the accuracy of grain boundary reconstruction but also significantly enhances the generalization ability of the network. This provides reliable technical support for the characterization of the microstructure and the performance prediction of metal materials.

Bi‐LORA: A Vision‐Language Approach for Synthetic Image Detection

ABSTRACTAdvancements in deep image synthesis techniques, such as generative adversarial networks (GANs) and diffusion models (DMs), have ushered in an era of generating highly realistic images. While this technological progress has captured significant interest, it has also raised concerns about the high challenge in distinguishing real images from their synthetic counterparts. This paper takes inspiration from the potent convergence capabilities between vision and language, coupled with the zero‐shot nature of vision‐language models (VLMs). We introduce an innovative method called Bi‐LORA that leverages VLMs, combined with low‐rank adaptation (LORA) tuning techniques, to enhance the precision of synthetic image detection for unseen model‐generated images. The pivotal conceptual shift in our methodology revolves around reframing binary classification as an image captioning task, leveraging the distinctive capabilities of cutting‐edge VLM, notably bootstrapping language image pre‐training (BLIP)2. Rigorous and comprehensive experiments are conducted to validate the effectiveness of our proposed approach, particularly in detecting unseen diffusion‐generated images from unknown diffusion‐based generative models during training, showcasing robustness to noise, and demonstrating generalisation capabilities to GANs. The experiments show that Bi‐LORA outperforms state of the art models in cross‐generator tasks because it leverages multi‐modal learning, open‐world visual knowledge, and benefits from robust, high‐level semantic understanding. By combining visual and textual knowledge, it can handle variations in the data distribution (such as those caused by different generators) and maintain strong performance across different domains. Its ability to transfer knowledge, robustly extract features and perform zero‐shot learning also contributes to its generalisation capabilities, making it more adaptable to new generators. The experimental results showcase an impressive average accuracy of 93.41% in synthetic image detection on unseen generation models. The code and models associated with this research can be publicly accessed at https://github.com/Mamadou‐Keita/VLM‐DETECT.

MSSA: multi-stage semantic-aware neural network for binary code similarity detection.

Binary code similarity detection (BCSD) aims to identify whether a pair of binary code snippets is similar, which is widely used for tasks such as malware analysis, patch analysis, and clone detection. Current state-of-the-art approaches are based on Transformer, which require substantial computation resources. Learning-based approaches remains room for optimization in learning the deeper semantics of binary code. In this paper, we propose MSSA, a multi-stage semantic-aware neural network for BCSD at the function level. It effectively integrates the semantic and structural information of assembly instructions within and between basic blocks, and across the entire function through four semantic-aware neural networks, achieving deep understanding of binary code semantics. MSSA is a lightweight model with only 0.38M parameters in its backbone network, suitable for deployment in CPU environments. Experimental results show that MSSA outperforms Gemini, Asm2Vec, SAFE, and jTrans in classification performance and ranks second only to the Transformer-based jTrans in retrieval performance.

Semantic scene understanding through advanced object context analysis in image

Semantic scene understanding through advanced object context analysis in image

Multilingual feasibility of GPT-4o for automated Voice-to-Text CT and MRI report transcription.

Large language models (LLMs) promise to streamline radiology reporting. With the release of OpenAI's GPT-4o (Generative Pre-trained Transformers-4 omni), which processes not only text but also speech, multimodal LLMs might now also be used as medical speech recognition software for radiology reporting in multiple languages. This proof-of-concept study investigates the feasibility of using GPT-4o for automated voice-to-text transcription of radiology reports in English and German. Three readers with varying levels of experience each dictated 100 synthetic radiology reports in both languages using GPT-4o via the ChatGPT iOS mobile application. Reports included CT and MRI scans of various anatomical regions. Evaluation metrics included error type, severity, and correction time. BERTScore and ROUGE metrics were calculated to assess semantic similarity and n-gram overlap between dictated and original reports. No significant differences in correction time between languages were found, but differences were observed between readers based on experience. Error rates were similar for both languages, with most errors being minor (92.68%, n=114/123 German; 94.74%, n=90/95 English) and technical (27.04%, n=43/159 German; 35.65%, n=41/115 English) or typographical (23.9%, n=38/159 German; 27.83%, n=32/115 English). BERTScore metrics were significantly higher for German, while ROUGE metrics showed no significant differences between languages. This study demonstrates the potential of GPT-4o for multilingual transcription of radiology reports, effectively handling both English and German with minimal errors and high semantic understanding. Future research should compare GPT-4o with current radiology dictation tools, assessing performance, cost-effectiveness, and multilingual capabilities across diverse speaker populations.

Integrating trajectory data and demographic characteristics: a trajectory semantic model for predicting travel flow and conducting interaction analysis

ABSTRACT With urbanisation and population growth, understanding spatial interactions in cities is increasingly vital for urban management. In recent decades, spatial interactions could be predicted accurately with the support of large GPS data, but anonymous trajectory data lacks semantic details, limiting predictions and behaviour understanding. To address this, we proposed a Semantic-Integrated Mobility Trajectory Model (SMTM), integrating social media check-in data, remote sensing imagery, and taxi trajectory data capable of accurately predict travel flow. Specifically, Convolutional Neural Networks (CNN) and Long Short Term Memory (LSTM) extract demographic insights, Graph Convolutional Networks (GCN) and Gate Recurrent Units (GRU) are incorporated to predict spatial interaction intensity. We conducted two case studies in New York City, U.S., and Ningbo, China, using taxi trips (over three million trips in New York and nearly one million trips in Ningbo) and social media check-in data (around 60,000 records for each city). Results demonstrate excellent performance over baselines. Furthermore, the integration of travel trajectories and census data revealed diverse travel preferences at various scales, including intra-region, inter-region, and inter-urban. The SMTM model contributes to optimising the design of public spaces and personalised recommendations.

Using colourful semantic approaches as a group intervention within primary schools to improve language development: A mixed methods design

Colourful Semantics is an approach initially designed to support language development in children within clinical settings (Bryan, 1997). While initial research was small and based on case study methodology, the research base is growing to include larger numbers of participants and populations. Studies in this area have begun to explore how impactful Colourful Semantic approaches are within school settings when delivered by teaching practitioners (Hettiarachchi, 2016; Hettiararchchi & Ranaweera, 2019). The current small scale study explores the impact of Colourful Semantic approaches when delivered by Additional Support for Learning teachers within small groups in primary schools. A mixed methods design is utilised with both standardised (Renfrew, 1997) and non-standardised pre and post assessments being employed. Additionally, practitioner experiences were gathered through an online questionnaire. While the standardised assessment scores presented mixed findings, closer analysis of narrative responses revealed improvements in sentence structure and organisation. Triangulation and analysis of non-standardised assessment scores and practitioner experiences, revealed significant improvements in both semantic understanding and mean utterance length. Additional benefits in cognitive skills were highlighted by practitioners. Limitations and future research to explore implementing Colourful Semantic approaches within school settings is explored.

PCB CT image element segmentation model based on boundary-attention-guided finetuning

Background Computed Tomography (CT) technology is commonly used to realize non-destructive testing of Printed Circuit Board (PCB), and element segmentation is the key link in the process. Although the pretraining and finetuning paradigm alleviates the problem of labeling, PCB CT images are easily affected by uneven grayscale and layer penetration. This leads to difficult segmentation of boundaries and affect semantic understanding, resulting in jagged boundaries and even missing elements. Objective This paper aims to solve the problem of poor boundary segmentation in PCB CT image element segmentation. Methods To this end, we propose PCB CT image element segmentation model based on boundary-attention-guided finetuning. An improved boundary detection algorithm is proposed to enhance boundary sensing ability. In order to achieve non-fixed weight feature fusion, the Attention Feature Fusion module is designed to help boundary features better assist segmentation through attention mechanism. Results Experiments show that BAG-FTseg can achieve 89.5% mIoU on our PCB CT dataset, exceeding the baseline model by 0.9%, and the boundary-mIoU reaches 69.5%, 5.3% higher than the baseline model. Conclusion This method improves the accuracy of boundary segmentation of PCB elements and the efficiency of feature fusion through attention mechanism, which has practical significance.

MGFusion: a multimodal large language model-guided information perception for infrared and visible image fusion

Existing image fusion methods primarily focus on complex network structure designs while neglecting the limitations of simple fusion strategies in complex scenarios. To address this issue, this study proposes a new method for infrared and visible image fusion based on a multimodal large language model. The method proposed in this paper fully considers the high demand for semantic information in enhancing image quality as well as the fusion strategies in complex scenes. We supplement the features in the fusion network with information from the multimodal large language model and construct a new fusion strategy. To achieve this goal, we design CLIP-driven Information Injection (CII) approach and CLIP-guided Feature Fusion (CFF) strategy. CII utilizes CLIP to extract robust image features rich in semantic information, which serve to supplement the information of infrared and visible features, thereby enhancing their representation capabilities for the scene. CFF further utilizes the robust image features extracted by CLIP to select and fuse the infrared and visible features after the injection of semantic information, addressing the challenges of image fusion in complex scenes. Compared to existing methods, the main advantage of the proposed method lies in leveraging the powerful semantic understanding capabilities of the multimodal large language model to supplement information for infrared and visible features, thus avoiding the need for complex network structure designs. Experimental results on multiple public datasets validate the effectiveness and superiority of the proposed method.

Speech Emotion Recognition Using Multi-Scale Global–Local Representation Learning with Feature Pyramid Network

Speech emotion recognition (SER) is important in facilitating natural human–computer interactions. In speech sequence modeling, a vital challenge is to learn context-aware sentence expression and temporal dynamics of paralinguistic features to achieve unambiguous emotional semantic understanding. In previous studies, the SER method based on the single-scale cascade feature extraction module could not effectively preserve the temporal structure of speech signals in the deep layer, downgrading the sequence modeling performance. To address these challenges, this paper proposes a novel multi-scale feature pyramid network. The enhanced multi-scale convolutional neural networks (MSCNNs) significantly improve the ability to extract multi-granular emotional features. Experimental results on the IEMOCAP corpus demonstrate the effectiveness of the proposed approach, achieving a weighted accuracy (WA) of 71.79% and an unweighted accuracy (UA) of 73.39%. Furthermore, on the RAVDESS dataset, the model achieves an unweighted accuracy (UA) of 86.5%. These results validate the system’s performance and highlight its competitive advantage.

Adapting SAM2 Model from Natural Images for Tooth Segmentation in Dental Panoramic X-Ray Images.

Dental panoramic X-ray imaging, due to its high cost-effectiveness and low radiation dose, has become a widely used diagnostic tool in dentistry. Accurate tooth segmentation is crucial for lesion analysis and treatment planning, helping dentists to quickly and precisely assess the condition of teeth. However, dental X-ray images often suffer from noise, low contrast, and overlapping anatomical structures, coupled with limited available datasets, leading traditional deep learning models to experience overfitting, which affects generalization ability. In addition, high-precision deep models typically require significant computational resources for inference, making deployment in real-world applications challenging. To address these challenges, this paper proposes a tooth segmentation method based on the pre-trained SAM2 model. We employ adapter modules to fine-tune the SAM2 model and introduce ScConv modules and gated attention mechanisms to enhance the model's semantic understanding and multi-scale feature extraction capabilities for medical images. In terms of efficiency, we utilize knowledge distillation, using the fine-tuned SAM2 model as the teacher model for distilling knowledge to a smaller model named LightUNet. Experimental results on the UFBA-UESC dataset show that, in terms of performance, our model significantly outperforms the traditional UNet model in multiple metrics such as IoU, effectively improving segmentation accuracy and model robustness, particularly with limited sample datasets. In terms of efficiency, LightUNet achieves comparable performance to UNet, but with only 1.6% of its parameters and 24.0% of the inference time, demonstrating its feasibility for deployment on edge devices.