Semantic Gap Research Articles

A novel network for semantic segmentation of landslide areas in remote sensing images with multi-branch and multi-scale fusion

Landslides pose significant risks as natural disasters, highlighting the importance of accurate mapping using remote sensing images for various practical applications. However, due to the challenges arising from incomplete and inaccurate boundary information of foreground landslide polygons, existing methods can only achieve suboptimal performance. To this premise, in this paper, we propose a segmentation network called GMNet that leverages global information extraction and multi-scale feature fusion to enhance the discrimination of landslides from other objects. Specifically, by employing a multi-branch mechanism, our method effectively captures global information, while an improved multi-scale feature fusion technique addresses the issue of varying scales in landslide polygons. Furthermore, semantic enhancement enhances the semantic information of low-level features, bridging the semantic gap and enhancing fusion efficacy. Experimental results demonstrate the effectiveness of our network in segmenting landslide areas accurately within the remote sensing image dataset. Especially, our F1_scores on three benchmarks outperform existing runner-ups by notable margins of 4.81%, 1.72%, and 1.16%, showcasing the value of our method in this domain.

Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer

Never having seen an object and heard its sound simultaneously, can the model still accurately localize its visual position from the input audio? In this work, we concentrate on the Audio-Visual Localization and Segmentation tasks but under the demanding zero-shot and few-shot scenarios. To achieve this goal, different from existing approaches that mostly employ the encoder-fusion-decoder paradigm to decode localization information from the fused audio-visual feature, we introduce the encoder-prompt-decoder paradigm, aiming to better fit the data scarcity and varying data distribution dilemmas with the help of abundant knowledge from pre-trained models. Specifically, we first propose to construct a Semantic-aware Audio Prompt (SAP) to help the visual foundation model focus on sounding objects, meanwhile, the semantic gap between the visual and audio modalities is also encouraged to shrink. Then, we develop a Correlation Adapter (ColA) to keep minimal training efforts as well as maintain adequate knowledge of the visual foundation model. By equipping with these means, extensive experiments demonstrate that this new paradigm outperforms other fusion-based methods in both the unseen class and cross-dataset settings. We hope that our work can further promote the generalization study of Audio-Visual Localization and Segmentation in practical application scenarios. Project page: https://github.com/GeWu-Lab/Generalizable-Audio-Visual-Segmentation

Mitigating Idiom Inconsistency: A Multi-Semantic Contrastive Learning Method for Chinese Idiom Reading Comprehension

Chinese idioms pose a significant challenge for machine reading comprehension due to their metaphorical meanings often diverging from their literal counterparts, leading to metaphorical inconsistency. Furthermore, the same idiom can have different meanings in different contexts, resulting in contextual inconsistency. Although deep learning-based methods have achieved some success in idioms reading comprehension, existing approaches still struggle to accurately capture idiom representations due to metaphorical inconsistency and contextual inconsistency of idioms. To address these challenges, we propose a novel model, Multi-Semantic Contrastive Learning Method (MSCLM), which simultaneously addresses metaphorical inconsistency and contextual inconsistency of idioms. To mitigate metaphorical inconsistency, we propose a metaphor contrastive learning module based on the prompt method, bridging the semantic gap between literal and metaphorical meanings of idioms. To mitigate contextual inconsistency, we propose a multi-semantic cross-attention module to explore semantic features between different metaphors of the same idiom in various contexts. Our model has been compared with multiple current latest models (including GPT-3.5) on multiple Chinese idiom reading comprehension datasets, and the experimental results demonstrate that MSCLM outperforms state-of-the-art models.

A Dual-Way Enhanced Framework from Text Matching Point of View for Multimodal Entity Linking

Multimodal Entity Linking (MEL) aims at linking ambiguous mentions with multimodal information to entity in Knowledge Graph (KG) such as Wikipedia, which plays a key role in many applications. However, existing methods suffer from shortcomings, including modality impurity such as noise in raw image and ambiguous textual entity representation, which puts obstacles to MEL. We formulate multimodal entity linking as a neural text matching problem where each multimodal information (text and image) is treated as a query, and the model learns the mapping from each query to the relevant entity from candidate entities. This paper introduces a dual-way enhanced (DWE) framework for MEL: (1) our model refines queries with multimodal data and addresses semantic gaps using cross-modal enhancers between text and image information. Besides, DWE innovatively leverages fine-grained image attributes, including facial characteristic and scene feature, to enhance and refine visual features. (2)By using Wikipedia descriptions, DWE enriches entity semantics and obtains more comprehensive textual representation, which reduces between textual representation and the entities in KG. Extensive experiments on three public benchmarks demonstrate that our method achieves state-of-the-art (SOTA) performance, indicating the superiority of our model. The code is released on https://github.com/season1blue/DWE.

CariesXrays: Enhancing Caries Detection in Hospital-Scale Panoramic Dental X-rays via Feature Pyramid Contrastive Learning

Dental caries has been widely recognized as one of the most prevalent chronic diseases in the field of public health. Despite advancements in automated diagnosis across various medical domains, it remains a substantial challenge for dental caries detection due to its inherent variability and intricacies. To bridge this gap, we release a hospital-scale panoramic dental X-ray benchmark, namely “CariesXrays”, to facilitate the advancements in high-precision computer-aided diagnosis for dental caries. It comprises 6,000 panoramic dental X-ray images, with a total of 13,783 instances of dental caries, all meticulously annotated by dental professionals. In this paper, we propose a novel Feature Pyramid Contrastive Learning (FPCL) framework, that jointly incorporates feature pyramid learning and contrastive learning within a unified diagnostic paradigm for automated dental caries detection. Specifically, a robust dual-directional feature pyramid network (D2D-FPN) is designed to adaptively capture rich and informative contextual information from multi-level feature maps, thus enhancing the generalization ability of caries detection across different scales. Furthermore, our model is augmented with an effective proposals-prototype contrastive regularization learning (P2P-CRL) mechanism, which can flexibly bridge the semantic gaps among diverse dental caries with varying appearances, resulting in high-quality dental caries proposals. Extensive experiments on our newly-established CariesXrays benchmark demonstrate the potential of FPCL to make a significant social impact on caries diagnosis.

Text-Based Occluded Person Re-identification via Multi-Granularity Contrastive Consistency Learning

Text-based Person Re-identification (T-ReID), which aims at retrieving a specific pedestrian image from a collection of images via text-based information, has received significant attention. However, previous research has overlooked a challenging yet practical form of T-ReID: dealing with image galleries mixed with occluded and inconsistent personal visuals, instead of ideal visuals with a full-body and clear view. Its major challenges lay in the insufficiency of benchmark datasets and the enlarged semantic gap incurred by arbitrary occlusions and modality gap between text description and visual representation of the target person. To alleviate these issues, we first design an Occlusion Generator (OGor) for the automatic generation of artificial occluded images from generic surveillance images. Then, a fine-granularity token selection mechanism is proposed to minimize the negative impact of occlusion for robust feature learning, and a novel multi-granularity contrastive consistency alignment framework is designed to leverage intra-/inter-granularity of visual-text representations for semantic alignment of occluded visuals and query texts. Experimental results demonstrate that our method exhibits superior performance. We believe this work could inspire the community to investigate more dedicated designs for implementing T-ReID in real-world scenarios. The source code is available at https://github.com/littlexinyi/MGCC.

A Unified Visual and Linguistic Semantics Method for Enhanced Image Captioning

Image captioning, also recognized as the challenge of transforming visual data into coherent natural language descriptions, has persisted as a complex problem. Traditional approaches often suffer from semantic gaps, wherein the generated textual descriptions lack depth, context, or the nuanced relationships contained within the images. In an effort to overcome these limitations, we introduce a novel encoder–decoder framework called A Unified Visual and Linguistic Semantics Method. Our method comprises three key components: an encoder, a mapping network, and a decoder. The encoder employs a fusion of CLIP (Contrastive Language–Image Pre-training) and SegmentCLIP to process and extract salient image features. SegmentCLIP builds upon CLIP’s foundational architecture by employing a clustering mechanism, thereby enhancing the semantic relationships between textual and visual elements in the image. The extracted features are then transformed by a mapping network into a fixed-length prefix. A GPT-2-based decoder subsequently generates a corresponding Chinese language description for the image. This framework aims to harmonize feature extraction and semantic enrichment, thereby producing more contextually accurate and comprehensive image descriptions. Our quantitative assessment reveals that our model exhibits notable enhancements across the intricate AIC-ICC, Flickr8k-CN, and COCO-CN datasets, evidenced by a 2% improvement in BLEU@4 and a 10% uplift in CIDEr scores. Additionally, it demonstrates acceptable efficiency in terms of simplicity, speed, and reduction in computational burden.

Bioethics-CSR Divide

Bioethics-CSR Divide

Lameness detection system for dairy cows based on instance segmentation

Lameness is one of the major health problems on dairy farms, which seriously affects dairy cow welfare and increases the risk of premature culling. Accuracy lameness detection ensures timely treatment of hooves and improves the level of health management on dairy farms. Most of the existing lameness detection methods detect one dairy cow, and it is difficult to detect lameness in multiple dairy cows at the same time. In this paper, a lameness detection system based on instance segmentation is constructed to extract the lameness features of multiple dairy cows and automatically detect lameness. First, an improved SOLOv2 network is designed to reduce the semantic gap between low-level and high-level features and improve the precision of dairy cow segmentation. Second, individual matching of dairy cows in different video frames is performed using the Hungarian algorithm. Third, the Canny algorithm is used to extract the neck and back contour features and key gait features of dairy cows. Finally, a deep learning model is constructed, and multiple lameness features are fused to detect the lameness of dairy cows. To evaluate the performance of the constructed system, 172 videos were randomly selected from 246 videos as training videos, and the remaining 74 videos were selected as the test videos. The lameness detection accuracy of the constructed system was 98.65%. The experimental results showed that the constructed system can extract the lameness features of multiple dairy cows at the same time and accurately detect the lameness of dairy cows.

Hyperspectral image classification method based on narrowing semantic gap convolutional neural network

ABSTRACT In recent years, numerous hyperspectral image classification methods based on convolutional neural network (CNN) have been proposed in recent years. However, the majority of CNN-based classification models fail to take into account the semantic gap between shallow and deep features, and instead, just sum up or splice these two types of features, which leads to subpar feature fusion. To address the above problems, the paper proposes a hyperspectral image classification method based on Narrowing Semantic Gap Convolutional Neural Network (NSGCNN). We developed the feature fusion sub-network, which consists of Feature Semantic Enhancement Module (FSEM), Spatial Detail Supplement Module (SDCM), and Cross-layer Feature Fusion Module (CFFM). FSEM extracts rich global semantic information based on multi-scale dilated convolution. Reverse attention mechanism is introduced in the SDCM to complement deep-layer features with fine spatial features like edges and textures. The fusing of hierarchical features is then achieved using CFFM. Additionally, we can regulate the model’s parameter count by regulating the number of output feature maps. Experimental results on three benchmark hyperspectral datasets demonstrate that NSGCNN outperforms other state-of-the-art methods in terms of classification performance and significantly outperforms them in terms of model complexity.

A cold-start recommendation method for We-map considering co-occurrence collaborative signals and contrastive learning

ABSTRACT We-maps, augmenting traditional maps, confront cold-start problems with prevailing recommendation techniques. The Contrastive Collaborative Filtering (CCF) model tackles We-map's cold-start issues by harnessing co-occurrence signals from warm data in a contrastive framework, thereby refining embeddings through the integration of content and co-occurrence insights. This methodology engenders unique embeddings for We-map entities, utilizing contrastive learning to bridge semantic gaps and enable indirect knowledge transfer, effectively mitigating cold-start challenges. Experimental validation on We-map and various public datasets confirms the method's practicality in addressing cold-start recommendations for We-maps. The study offers a new path for efficient geospatial data dissemination.

ProMvSD: Towards unsupervised knowledge graph anomaly detection via prior knowledge integration and multi-view semantic-driven estimation

Knowledge graphs (KGs) have found extensive applications within intelligent systems, such as information retrieval. Much of the research has predominantly focused on completing missing knowledge, with little consideration given to examining errors. Unfortunately, during customizing KGs, diverse unpredictable errors are virtually unavoidable to be introduced, and these anomalies significantly impact the performance of applications. Detecting erroneous knowledge presents a formidable challenge due to the costly acquisition of ground-truth labels. In this work, we develop an unsupervised anomaly detection framework named ProMvSD, aiming to adapt KGs of varying scales via serialization components. To overcome the insufficient contextual information provided by the topological structure, we introduce the large language model as a reasoner to extract prior knowledge from extensive pre-trained textual data, thereby enhancing the understanding of KGs. Anomalous triple may result in a larger semantic gap between the head and tail neighborhoods. To uncover latent anomalies effectively, we propose a multi-view semantic-driven model (MvSD) based on the assumptions of self-consistency and information stability. MvSD jointly estimates the suspiciousness of triples from three hyperviews: node-view semantic contradiction, triple-view semantic gap, and pathway-view semantic gap. Extensive experiments on three English benchmark KGs and a Chinese medical KG demonstrate that, for the top 1% of the most suspicious triples, we can detect real anomalies with at most 99.9% accuracy. Furthermore, ProMvSD significantly outperforms state-of-the-art representation learning baselines, achieving a 29.2% improvement in detecting all anomalies.

Time-series land cover change detection using deep learning-based temporal semantic segmentation

The process of sustainable urban development is accompanied by frequent and complex land cover changes, and thus, clarify accurate information on land cover changes can provide scientific data for urban management. To characterize urban development at an accurate spatiotemporal scale, a change detection model is not only required to provide accurate location (Where) and time (When) of the changes, but also semantic information on the change types (What). Accordingly, this study proposed a deep learning method for temporal semantic segmentation change detection (TSSCD) that obtains information on the where, when, and what of changes simultaneously. TSSCD model bridges the semantic gap between remote sensing time series abrupt changes and land cover changes by learning the month-to-month mapping from spectral information to land cover types. We implemented a temporal semantic segmentation model based on the most classic fully convolutional network, where all two-dimensional convolutions and pooling operations were replaced with one-dimensional. We conducted tests on the TSSCD in several urban study areas, and it consistently exhibited good accuracy. In most cases, it outperformed the BFAST and CCDC algorithms, except when only a single spectral band was used. Simultaneously, we analyzed the minimum data requirements for training a TSSCD. The TSSCD currently faces challenges in achieving strong generalization beyond the training data distribution. Additionally, we observed that change detection for specific land cover types can be achieved through the flexible configuration of TSSCD. Finally, we explored a method for constructing datasets using existing products to minimize data annotation efforts, yielding promising results. However, there is still some gap compared to complete manual annotation. Overall, the TSSCD model provided a novel solution to accurately characterize sustainable urban development at the spatiotemporal scale.

Dual teachers for self-knowledge distillation

We introduce an efficient self-knowledge distillation framework, Dual Teachers for Self-Knowledge Distillation (DTSKD), where the student receives self-supervisions by dual teachers from two substantially different fields, i.e., the past learning history and the current network structure. Specifically, DTSKD trains a considerably lightweight multi-branch network and acquires predictions from each, which are simultaneously supervised by a historical teacher from the previous epoch and a structural teacher under the current iteration. To our best knowledge, it is the first attempt to jointly conduct historical and structural self-knowledge distillation in a unified framework where they demonstrate complementary and mutual benefits. The Mixed Fusion Module (MFM) is further developed to bridge the semantic gap between deep stages and shallow branches by iteratively fusing multi-stage features based on the top-down topology. Extensive experiments prove the effectiveness of our proposed method, showing superior performance over related state-of-the-art self-distillation works on three datasets: CIFAR-100, ImageNet-2012, and PASCAL VOC.

Visual Clue Guidance and Consistency Matching Framework for Multimodal Named Entity Recognition

The goal of multimodal named entity recognition (MNER) is to detect entity spans in given image–text pairs and classify them into corresponding entity types. Despite the success of existing works that leverage cross-modal attention mechanisms to integrate textual and visual representations, we observe three key issues. Firstly, models are prone to misguidance when fusing unrelated text and images. Secondly, most existing visual features are not enhanced or filtered. Finally, due to the independent encoding strategies employed for text and images, a noticeable semantic gap exists between them. To address these challenges, we propose a framework called visual clue guidance and consistency matching (GMF). To tackle the first issue, we introduce a visual clue guidance (VCG) module designed to hierarchically extract visual information from multiple scales. This information is utilized as an injectable visual clue guidance sequence to steer text representations for error-insensitive prediction decisions. Furthermore, by incorporating a cross-scale attention (CSA) module, we successfully mitigate interference across scales, enhancing the image’s capability to capture details. To address the third issue of semantic disparity between text and images, we employ a consistency matching (CM) module based on the idea of multimodal contrastive learning, facilitating the collaborative learning of multimodal data. To validate the effectiveness of our proposed framework, we conducted comprehensive experimental studies, including extensive comparative experiments, ablation studies, and case studies, on two widely used benchmark datasets, demonstrating the efficacy of the framework.

Bridging the Cross-Modality Semantic Gap in Visual Question Answering.

The objective of visual question answering (VQA) is to adequately comprehend a question and identify relevant contents in an image that can provide an answer. Existing approaches in VQA often combine visual and question features directly to create a unified cross-modality representation for answer inference. However, this kind of approach fails to bridge the semantic gap between visual and text modalities, resulting in a lack of alignment in cross-modality semantics and the inability to match key visual content accurately. In this article, we propose a model called the caption bridge-based cross-modality alignment and contrastive learning model (CBAC) to address the issue. The CBAC model aims to reduce the semantic gap between different modalities. It consists of a caption-based cross-modality alignment module and a visual-caption (V-C) contrastive learning module. By utilizing an auxiliary caption that shares the same modality as the question and has closer semantic associations with the visual, we are able to effectively reduce the semantic gap by separately matching the caption with both the question and the visual to generate pre-alignment features for each, which are then used in the subsequent fusion process. We also leverage the fact that V-C pairs exhibit stronger semantic connections compared to question-visual (Q-V) pairs to employ a contrastive learning mechanism on visual and caption pairs to further enhance the semantic alignment capabilities of single-modality encoders. Extensive experiments conducted on three benchmark datasets demonstrate that the proposed model outperforms previous state-of-the-art VQA models. Additionally, ablation experiments confirm the effectiveness of each module in our model. Furthermore, we conduct a qualitative analysis by visualizing the attention matrices to assess the reasoning reliability of the proposed model.

On Ambiguity in the Arabic Language: Scrutinizing Translation Issues through Machine Translation from English and French into Arabic

Ambiguity in some syntactic structures of the same language has always posed problems to the human translator and to machine translation. These problems become more complex for the Machine Translation of genetically unrelated languages such as Arabic, English and French. Arabic Lexical ambiguity in Natural Language Processing (NLP) also poses problems when the semantic fields of Arabic words differ from those of English for instance. This often occurs when two or more words from Arabic equate to a single word in English. Semantic gaps between the two languages are also a source of ambiguity in Natural Language Processing. We shall deal with some cases of ambiguity in machine translation from Arabic to English and French and vice versa. The questions addressed in this paper relate to segmentation, determination / non-determination, coordination and the issue of the word as a meaningful and functional unit. Some aspects of the segmentation of constituents into grammatical categories and their comparison with structures of Arabic English and French are addressed in this paper.

DI-VTR: Dual inter-modal interaction model for video-text retrieval

Video-text retrieval is a challenging task for multimodal information processing due to the semantic gap between different modalities. However, most existing methods do not fully mine the intra-modal interactions, as with the temporal correlation of video frames, which results in poor matching performance. Additionally, the imbalanced semantic information between videos and texts also leads to difficulty in the alignment of the two modalities. To this end, we propose a dual inter-modal interaction network for video-text retrieval, i.e., DI-VTR. To learn the intra-modal interaction of video frames, we design a contextual-related video encoder to obtain more fine-grained content-oriented video representations. We also propose a dual inter-modal interaction module to accomplish accurate multilingual alignment between the video and text modalities by introducing multilingual text to improve the representation ability of text semantic features. Extensive experimental results on commonly-used video-text retrieval datasets, including MSR-VTT, MSVD and VATEX, show that the proposed method achieves significantly improved performance compared with state-of-the-art methods.

Multi-level textual-visual alignment and fusion network for multimodal aspect-based sentiment analysis

Multimodal Aspect-Based Sentiment Analysis (MABSA) is an essential task in sentiment analysis that has garnered considerable attention in recent years. Typical approaches in MABSA often utilize cross-modal Transformers to capture interactions between textual and visual modalities. However, bridging the semantic gap between modalities spaces and addressing interference from irrelevant visual objects at different scales remains challenging. To tackle these limitations, we present the Multi-level Textual-Visual Alignment and Fusion Network (MTVAF) in this work, which incorporates three auxiliary tasks. Specifically, MTVAF first transforms multi-level image information into image descriptions, facial descriptions, and optical characters. These are then concatenated with the textual input to form a textual+visual input, facilitating comprehensive alignment between visual and textual modalities. Next, both inputs are fed into an integrated text model that incorporates relevant visual representations. Dynamic attention mechanisms are employed to generate visual prompts to control cross-modal fusion. Finally, we align the probability distributions of the textual input space and the textual+visual input space, effectively reducing noise introduced during the alignment process. Experimental results on two MABSA benchmark datasets demonstrate the effectiveness of the proposed MTVAF, showcasing its superior performance compared to state-of-the-art approaches. Our codes are available at https://github.com/MKMaS-GUET/MTVAF.

Deep Learning-Based Medical Image Registration Algorithm: Enhancing Accuracy with Dense Connections and Channel Attention Mechanisms

In critical clinical medical image analysis applications, such as surgical navigation and tumor monitoring, image registration is crucial. Recognizing the potential for enhanced accuracy in existing unsupervised image registration techniques for single-modal imagery, this research introduces an innovative deep learning-based image registration algorithm. Its novelty resides in integrating short and long connections to create a densely connected structure, markedly refining the feature map interconnectivity within the U-Net architecture. This advancement addresses the significant semantic gap issues arising from disparities in feature map sampling depths. Moreover, the algorithm incorporates a channel attention mechanism within the U-shaped network's decoder, significantly mitigating image noise and facilitating the generation of smoother deformation fields. This enhancement not only boosts the model's detail sensitivity but also markedly increases image registration precision, particularly evident when processing single-modal brain MRI datasets, thereby proving the algorithm's efficacy and utility. Extensive clinical application-based training and testing have underscored this algorithm's substantial contributions to medical image registration accuracy enhancement. Overall, by leveraging deep learning technologies and innovative algorithmic structures, this study addresses pivotal challenges in medical image registration, offering more precise and dependable support for clinical applications like surgical navigation and tumor surveillance.