Modal Gap Research Articles

Self-prompting semantic segmentation of bridge point cloud data using a large computer vision model

Semantic segmentation of bridge Point Cloud Data (PCD) is an intermediate process required for the tasks such as deformation detection and digital twin. However, existing methods either require a substantial amount of training data or exhibit limited generalization ability. To address these issues, this paper presents an unsupervised framework for semantic segmentation of bridge PCD. A visible point rendering method is first employed to bridge the modal gap between 2D and 3D, and then a self-prompting segmentation method based on a large computer vision model is introduced to achieve instance segmentation. Experiment results on the real-world reinforced concrete bridge dataset and suspension bridge dataset showed the proposed method achieved the outstanding performance on all overall evaluation metrics of overall accuracy (98.31 % and 98.10 %), mean precision (97.67 % and 93.96 %), mean recall rate (97.50 % and 97.51 %) and mean F1 score (97.46 % and 95.55 %). The comparisons with existing methods demonstrate that without the need of training data, our method can achieve competitive or even superior accuracy to learning-based methods.

Visual primitives as words: Alignment and interaction for compositional zero-shot learning

Compositional Zero-Shot Learning (CZSL) aims to recognize seen and unseen attribute-object compositions. Recently, some researchers apply vision-language models to CZSL task. However, they only roughly match the image embedding and composition embedding on the image level, which can be a barrier to further improvement. With observation and analysis, we believe that a visual primitive is worth a word. To make full use of visual primitives to achieve fine-grained alignment and bridging modal gap, we propose VisPrompt for interacting visual primitives with sub-concepts in a prompt. Specifically, VisPrompt aligns the visual primitives (i.e., visual attribute and visual object) with the sub-concepts (i.e., text attribute and text object) at a fine-grained level. It consists of two steps: (1) First, we extract the visual attribute embedding by an attribute extraction module, and the visual object embedding by an object extraction module; (2) Second, we design an attribute-wise prompt, an object-wise prompt, and a visual reconstructed prompt to be encoded, where a visual primitive plays the role of corresponding sub-concept to interact. Therefore, our model is capable of applying fine-grained alignment and bridging the gap between vision and text. Sufficient experiments on widely-used MIT-States, UT-Zappos, CGQA, and VAW-CZSL datasets show that our VisPromt achieves state-of-the-art on the core metric AUC.

A Novel Energy Based Model Mechanism for Multi-Modal Aspect-Based Sentiment Analysis

Multi-modal aspect-based sentiment analysis (MABSA) has recently attracted increasing attention. The span-based extraction methods, such as FSUIE, demonstrate strong performance in sentiment analysis due to their joint modeling of input sequences and target labels. However, previous methods still have certain limitations: (i) They ignore the difference in the focus of visual information between different analysis targets (aspect or sentiment). (ii) Combining features from uni-modal encoders directly may not be sufficient to eliminate the modal gap and can cause difficulties in capturing the image-text pairwise relevance. (iii) Existing span-based methods for MABSA ignore the pairwise relevance of target span boundaries. To tackle these limitations, we propose a novel framework called DQPSA. Specifically, our model contains a Prompt as Dual Query (PDQ) module that uses the prompt as both a visual query and a language query to extract prompt-aware visual information and strengthen the pairwise relevance between visual information and the analysis target. Additionally, we introduce an Energy-based Pairwise Expert (EPE) module that models the boundaries pairing of the analysis target from the perspective of an Energy-based Model. This expert predicts aspect or sentiment span based on pairwise stability. Experiments on three widely used benchmarks demonstrate that DQPSA outperforms previous approaches and achieves a new state-of-the-art performance. The code will be released at https://github.com/pengts/DQPSA.

Autoencoder-Based Collaborative Attention GAN for Multi-Modal Image Synthesis

Multi-modal images are required in a wide range of practical scenarios, from clinical diagnosis to public security. However, certain modalities may be incomplete or unavailable because of the restricted imaging conditions, which commonly leads to decision bias in many real-world applications. Despite the significant advancement of existing image synthesis techniques, learning complementary information from multi-modal inputs remains challenging. To address this problem, we propose an autoencoder-based collaborative attention generative adversarial network (ACA-GAN) that uses available multi-modal images to generate the missing ones. The collaborative attention mechanism deploys a single-modal attention module and a multi-modal attention module to effectively extract complementary information from multiple available modalities. Considering the significant modal gap, we further developed an autoencoder network to extract the self-representation of target modality, guiding the generative model to fuse target-specific information from multiple modalities. This considerably improves cross-modal consistency with the desired modality, thereby greatly enhancing the image synthesis performance. Quantitative and qualitative comparisons for various multi-modal image synthesis tasks highlight the superiority of our approach over several prior methods by demonstrating more precise and realistic results. The source code will be available at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/caohaifang123/ACA-GAN</uri> .

Hybrid cross-modal interaction learning for multimodal sentiment analysis

Multimodal sentiment analysis (MSA) predicts the sentiment polarity of an unlabeled utterance that carries multiple modalities, such as text, vision and audio, by analyzing labeled utterances. Existing fusion methods mainly focus on establishing the relationship of characteristics among different modalities to enhance their emotion recognition abilities. However, they always ignore the all-round interaction between different modalities, especially the cross-modal interaction, which is critical to the sentiment decision of multimodal data. To address these issues, we propose a novel hybrid cross-modal interaction learning (HCIL) framework for hybrid learning of intra-modal, inter-modal, interactive-modal and cross-modal interactions, with which the model can fully utilize the sentiment information of multimodalities and enhance the sentiment assistance between modalities. Specifically, we propose two core substructures to learn discriminative multimodal features. One is the comparative learning interaction structure that can track the class dynamics in the intra-modal, reduce the modal gap in the inter-modal and establish emotional communication in the interactive-modal; the other is the cross-modal prediction structure, which can build the sentiment relationship between cross-modal pairs, especially exploring the auxiliary sentiment effect of audio on the vision and text. Furthermore, we adopt a hierarchical feature fusion structure to generate the multimodal feature for the final sentiment prediction. Extensive experiments on three benchmark datasets showed that our HCIL approach can obtain significant performance on the MSA task and that the design of a cross-modal interaction structure can directly promote the improvement of sentiment classification performance.

Similarity Graph-correlation Reconstruction Network for unsupervised cross-modal hashing

Existing cross-modal hash retrieval methods can simultaneously enhance retrieval speed and reduce storage space. However, these methods face a major challenge in determining the similarity metric between two modalities. Specifically, the accuracy of intra-modal and inter-modal similarity measurements is inadequate, and the large gap between modalities leads to semantic bias. In this paper, we propose a Similarity Graph-correlation Reconstruction Network (SGRN) for unsupervised cross-modal hashing. Particularly, the local relation graph rebasing module is used to filter out graph nodes with weak similarity and associate graph nodes with strong similarity, resulting in fine-grained intra-modal similarity relation graphs. The global relation graph reconstruction module is further strengthens cross-modal correlation and implements fine-grained similarity alignment between modalities. In addition, in order to bridge the modal gap, we combine the similarity representation of real-valued and hash features to design the intra-modal and inter-modal training strategies. SGRN conducted extensive experiments on two cross-modal retrieval datasets, and the experimental results effectively validated the superiority of the proposed method and significantly improved the retrieval performance.

Syncretic Space Learning Network for NIR-VIS Face Recognition

To overcome the technical bottleneck of face recognition in low-light scenarios, Near-InfraRed and VISible (NIR-VIS) heterogeneous face recognition is proposed for matching well-lit VIS faces with poorly lit NIR faces. Current cross-modal synthesis methods visually convert the NIR modality to the VIS modality and then perform face matching in the VIS modality. However, using a heavyweight GAN network on unpaired NIR-VIS faces may lead to high synthesis difficulty, low inference efficiency, and other problems. To alleviate the above problems, we simultaneously synthesize NIR and VIS images into modality-independent syncretic images and propose a novel syncretic space learning (SSL) model to eliminate the modal gap. First, Syncretic Modality Generator (SMG) synthesizes NIR and VIS images into syncretic images using channel-level convolution with a shallow CNN. In particular, the discriminative structural information is well preserved and the face quality can be further improved with small modal variations in a self-supervised learning manner. Second, Modality-adversarial Syncretic space Learning (MSL) projects NIR and VIS images into the syncretic space by a syncretic-modality adversarial learning strategy with syncretic pattern guided objective, so the modal gap of NIR-VIS faces can be effectively reduced. Finally, the Syncretic Distribution Consistency (SDC) constructed by NIR-syncretic, syncretic-syncretic, and VIS-syncretic consistency can enhance the intra-class compactness and learn discriminative representations. Extensive experiments on three challenging datasets demonstrate the effectiveness of the SSL method.

Visual Description Augmented Integration Network for Multimodal Entity and Relation Extraction

Multimodal Named Entity Recognition (MNER) and multimodal Relationship Extraction (MRE) play an important role in processing multimodal data and understanding entity relationships across textual and visual domains. However, irrelevant image information may introduce noise that misleads the recognition of information. Additionally, visual and semantic features originate from different modalities, and modal disparity hinders semantic alignment. Therefore, this paper proposes the Visual Description Augmentation Integration Network (VDAIN), which introduces an image description generation technique that allows semantic features generated from image descriptions to be presented in the same modality as the semantic features of textual information. This not only reduces the modal gap but also captures more accurately the high-level semantic information and underlying visual structure in the images. To filter out the modal noise, we use VDAIN to adaptively fuse visual features, semantic features of image descriptions, and textual information, thus eliminating irrelevant modal noise. The F1 score of the proposed model in this paper reaches 75.8% and 87.78% for the MNER task and 82.54% for the MRE task on the three public data sets, respectively, which are significantly better than the baseline model. The experimental results demonstrate the effectiveness of the proposed method in solving the modal noise and modal gap problems.

Survey of Cross-Modal Person Re-Identification from a Mathematical Perspective

Person re-identification (Re-ID) aims to retrieve a particular pedestrian’s identification from a surveillance system consisting of non-overlapping cameras. In recent years, researchers have begun to focus on open-world person Re-ID tasks based on non-ideal situations. One of the most representative of these is cross-modal person Re-ID, which aims to match probe data with target data from different modalities. According to the modalities of probe and target data, we divided cross-modal person Re-ID into visible–infrared, visible–depth, visible–sketch, and visible–text person Re-ID. In cross-modal person Re-ID, the most challenging problem is the modal gap. According to the different methods of narrowing the modal gap, we classified the existing works into picture-based style conversion methods, feature-based modality-invariant embedding mapping methods, and modality-unrelated auxiliary information mining methods. In addition, by generalizing the aforementioned works, we find that although deep-learning-based models perform well, the black-box-like learning process makes these models less interpretable and generalized. Therefore, we attempted to interpret different cross-modal person Re-ID models from a mathematical perspective. Through the above work, we attempt to compensate for the lack of mathematical interpretation of models in previous person Re-ID reviews and hope that our work will bring new inspiration to researchers.

Bridging the cross-modal gap using adversarial training for speech-to-text translation

The end-to-end speech translation (ST) model usually adopts the encoder-decoder structure, which takes the speech of the source language as input and directly outputs its translation result in the target language. Since the model performs cross-modal translation, it needs to extract semantically rich representations from speech. Compared with text, speech is fine-grained and contains more noise, which puts a great burden on the encoder of the ST model. The modal gap between speech and text usually causes the ST model to perform inferior to the corresponding machine translation (MT) model. To bridge the cross-modal gap, this paper proposes to use adversarial training to relieve the burden on the ST encoder by providing internal supervision signals. In this approach, the encoder in the ST model can extract representations that contain rich semantics, which greatly improves performance. Experiments on the datasets of Augmented Librispeech English-French and MuST-C English-German show the effectiveness of our method. Further analysis indicates that our proposed method can perform well in low-resource conditions compared to strong baselines.

Minimizing Maximum Feature Space Deviation for Visible-Infrared Person Re-Identification

Visible-infrared person re-identification (VIPR) has great potential for intelligent video surveillance systems at night, but it is challenging due to the huge modal gap between visible and infrared modalities. For that, this paper proposes a minimizing maximum feature space deviation (MMFSD) method for VIPR. First, this paper calculates visible and infrared feature centers of each identity. Second, this paper defines feature space deviations based on these feature centers to measure the modal gap between visible and infrared modalities. Third, this paper minimizes the maximum feature space deviation to significantly reduce the modal gap between visible and infrared modalities. Experimental results show the superiority of the proposed method, e.g., on the RegDB dataset, the rank-1 accuracy reaches 92.19%.

SIG-Former: monocular surgical instruction generation with transformers

Purpose:Automatic surgical instruction generation is a crucial part for intra-operative surgical assistance. However, understanding and translating surgical activities into human-like sentences are particularly challenging due to the complexity of surgical environment and the modal gap between images and natural languages. To this end, we introduce SIG-Former, a transformer-backboned generation network to predict surgical instructions from monocular RGB images.Methods:Taking a surgical image as input, we first extract its visual attentive feature map with a fine-tuned ResNet-101 model, followed by transformer attention blocks to correspondingly model its visual representation, text embedding and visual–textual relational feature. To tackle the loss-metric inconsistency between training and inference in sequence generation, we additionally apply a self-critical reinforcement learning approach to directly optimize the CIDEr score after regular training.Results:We validate our proposed method on DAISI dataset, which contains 290 clinical procedures from diverse medical subjects. Extensive experiments demonstrate that our method outperforms the baselines and achieves promising performance on both quantitative and qualitative evaluations.Conclusion:Our experiments demonstrate that SIG-Former is capable of mapping dependencies between visual feature and textual information. Besides, surgical instruction generation is still at its preliminary stage. Future works include collecting large clinical dataset, annotating more reference instructions and preparing pre-trained models on medical images.

TVT: Three-Way Vision Transformer through Multi-Modal Hypersphere Learning for Zero-Shot Sketch-Based Image Retrieval

In this paper, we study the zero-shot sketch-based image retrieval (ZS-SBIR) task, which retrieves natural images related to sketch queries from unseen categories. In the literature, convolutional neural networks (CNNs) have become the de-facto standard and they are either trained end-to-end or used to extract pre-trained features for images and sketches. However, CNNs are limited in modeling the global structural information of objects due to the intrinsic locality of convolution operations. To this end, we propose a Transformer-based approach called Three-Way Vision Transformer (TVT) to leverage the ability of Vision Transformer (ViT) to model global contexts due to the global self-attention mechanism. Going beyond simply applying ViT to this task, we propose a token-based strategy of adding fusion and distillation tokens and making them complementary to each other. Specifically, we integrate three ViTs, which are pre-trained on data of each modality, into a three-way pipeline through the processes of distillation and multi-modal hypersphere learning. The distillation process is proposed to supervise fusion ViT (ViT with an extra fusion token) with soft targets from modality-specific ViTs, which prevents fusion ViT from catastrophic forgetting. Furthermore, our method learns a multi-modal hypersphere by performing inter- and intra-modal alignment without loss of uniformity, which aims to bridge the modal gap between modalities of sketch and image and avoid the collapse in dimensions. Extensive experiments on three benchmark datasets, i.e., Sketchy, TU-Berlin, and QuickDraw, demonstrate the superiority of our TVT method over the state-of-the-art ZS-SBIR methods.

Semantic-guided autoencoder adversarial hashing for large-scale cross-modal retrieval

With the vigorous development of mobile Internet technology and the popularization of smart devices, while the amount of multimedia data has exploded, its forms have become more and more diversified. People’s demand for information is no longer satisfied with single-modal data retrieval, and cross-modal retrieval has become a research hotspot in recent years. Due to the strong feature learning ability of deep learning, cross-modal deep hashing has been extensively studied. However, the similarity of different modalities is difficult to measure directly because of the different distribution and representation of cross-modal. Therefore, it is urgent to eliminate the modal gap and improve retrieval accuracy. Some previous research work has introduced GANs in cross-modal hashing to reduce semantic differences between different modalities. However, most of the existing GAN-based cross-modal hashing methods have some issues such as network training is unstable and gradient disappears, which affect the elimination of modal differences. To solve this issue, this paper proposed a novel Semantic-guided Autoencoder Adversarial Hashing method for cross-modal retrieval (SAAH). First of all, two kinds of adversarial autoencoder networks, under the guidance of semantic multi-labels, maximize the semantic relevance of instances and maintain the immutability of cross-modal. Secondly, under the supervision of semantics, the adversarial module guides the feature learning process and maintains the modality relations. In addition, to maintain the inter-modal correlation of all similar pairs, this paper use two types of loss functions to maintain the similarity. To verify the effectiveness of our proposed method, sufficient experiments were conducted on three widely used cross-modal datasets (MIRFLICKR, NUS-WIDE and MS COCO), and compared with several representatives advanced cross-modal retrieval methods, SAAH achieved leading retrieval performance.

Clustering-driven Deep Adversarial Hashing for scalable unsupervised cross-modal retrieval

With the advent of the big data era, multimedia data is growing rapidly, and its data modalities is also becoming diversified. Therefore, the demand for the speed and accuracy of cross-modal information retrieval is increasing. Hashing-based cross-modal retrieval technology attracts widespread attention, it encodes multimedia data into a common binary hash space, thereby effectively measuring the correlation between samples from different modalities. In this paper, we propose a novel end-to-end deep cross-modal retrieval framework, namely Clustering-driven Deep Adversarial Hashing (CDAH), which has three main characteristics. Firstly, CDAH learns discriminative clusters recursively through a soft clustering model. It attempts to generate modal-invariant representations in a common space by obfuscating the modality classifier, which tries to distinguish different modalities according to the generated representations. Secondly, in order to minimize the modal gap between feature representations from different modalities with the same semantic label, and to maximize the distance between images and texts with different labels, CDAH constructs a fused-semantics matrix to integrate the original domain information from different modalities, serving as self-supervised information to refine the binary codes. Finally, CDAH skillfully uses a scaled tanh function to adaptively learn the binary codes, which will gradually converge to the original tricky binary coding problem. We conduct comprehensive experiments on four popular datasets, and the experimental results demonstrate the superiority of our model against the state-of-the-art methods.

Learning Modal-Invariant Angular Metric by Cyclic Projection Network for VIS-NIR Person Re-Identification.

Person re-identification across visible and near-infrared cameras (VIS-NIR Re-ID) has widespread applications. The challenge of this task lies in heterogeneous image matching. Existing methods attempt to learn discriminative features via complex feature extraction strategies. Nevertheless, the distributions of visible and near-infrared features are disparate caused by modal gap, which significantly affects feature metric and makes the performance of the existing models poor. To address this problem, we propose a novel approach from the perspective of metric learning. We conduct metric learning on a well-designed angular space. Geometrically, features are mapped from the original space to the hypersphere manifold, which eliminates the variations of feature norm and concentrates on the angle between the feature and the target category. Specifically, we propose a cyclic projection network (CPN) that transforms features into an angle-related space while identity information is preserved. Furthermore, we proposed three kinds of loss functions, AICAL, LAL and DAL, in angular space for angular metric learning. Multiple experiments on two existing public datasets, SYSU-MM01 and RegDB, show that performance of our method greatly exceeds the SOTA performance.

Semantic ranking structure preserving for cross-modal retrieval

Cross-modal retrieval not only needs to eliminate the heterogeneity of modalities, but also needs to constrain the return order of retrieval results. Accordingly, we propose a novel common representation space learning method, called Semantic Ranking Structure Preserving (SRSP) for Cross-modal Retrieval in this paper. First, the dependency relationship between labels is used to minimize the discriminative loss of multi-modal data and mine potential relationships between samples to get richer semantic information in the common space. Second, we constrain the correlation ranking of representations in common space, so as to break the modal gap and promote the multi-modal correlation learning. The comprehensive experimental comparison results show that our algorithm substantially enhances the performance and consistently outperforms very recent algorithms in terms of widely used cross-modal benchmark datasets.

High-Q microcavities in low-index one-dimensional photonic crystal slabs based on modal gap confinement

Recently, various high quality factor photonic crystal microcavities have been demonstrated theoretically and experimentally with only one-dimensional periodicity. However, in most cases high-index materials such as silicon were chosen for easily achievable large photonic bandgap and elaborate refractive index modulation or taper structure is required for reducing radiation loss. Here, we present a design of high-Q microcavities in one-dimensional multilayer polystyrene photonic crystal slab structures with a low-index contrast of 1.59:1. Microcavities are introduced by simply decreasing the thickness of layers at the center region to form a double-heterostructure. A resonant mode with a quality factor up to 20 000 is obtained and found to originate from the modal gap confinement by comparing with a Fabry–Perot cavity. The dependence of the maximal quality factor on the cavity length further reveals that the small group velocity of light within the heterostructure cavity contributes significantly to the high-Q. In terms of the high quality factor, ease of fabrication, and large Kerr nonlinearity of polystyrene, our double-heterostructure microcavities will find potential application in realizing all-optical modulation devices.

The modal gap: the objective problem of Lessing's ditch(es) and Kierkegaard's subjective reply

This essay expands upon the suggestion that Lessing's infamous ‘ditch’ is actually three ditches: temporal, metaphysical, and existential gaps. It examines the complex problems these ditches raise, and then proposes that Kierkegaard's Fragments and Postscript exhibit a similar triadic organizational structure, which may signal a deliberate attempt to engage and respond to Lessing's three gaps. Viewing the Climacean project in this way offers an enhanced understanding of the intricacies of Lessing's rationalist approach to both religion and historical truth, and illuminates Climacus's subjective response to Lessing.