Textual Modalities Research Articles

A global multimodal flood event dataset with heterogeneous text and multi-source remote sensing images

ABSTRACT With the increasing frequency of floods, in-depth flood event analyses are essential for effective disaster relief and prevention. Satellite-based flood event datasets have become the primary data source for flood event analyses instead of limited disaster maps due to their enhanced availability. Nevertheless, despite the vast amount of available remote sensing images, existing flood event datasets continue to pose significant challenges in flood event analyses due to the uneven geographical distribution of data, the scarcity of time series data, and the limited availability of flood-related semantic information. There has been a surge in acceptance of deep learning models for flood event analyses, but some existing flood datasets do not align well with model training, and distinguishing flooded areas has proven difficult with limited data modalities and semantic information. Moreover, efficient retrieval and pre-screening of flood-related imagery from vast satellite data impose notable obstacles, particularly within large-scale analyses. To address these issues, we propose a Multimodal Flood Event Dataset (MFED) for deep-learning-based flood event analyses and data retrieval. It consists of 18 years of multi-source remote sensing imagery and heterogeneous textual information covering flood-prone areas worldwide. Incorporating optical and radar imagery can exploit the correlation and complementarity between distinct image modalities to capture the pixel features in flood imagery. It is worth noting that text modality data, including auxiliary hydrological information extracted from the Global Flood Database and text information refined from online news records, can also offer a semantic supplement to the images for flood event retrieval and analysis. To verify the applicability of the MFED in deep learning models, we carried out experiments with different models using a single modality and different combinations of modalities, which fully verified the effectiveness of the dataset. Furthermore, we also verify the efficiency of the MFED in comparative experiments with existing multimodal datasets and diverse neural network structures.

Vision-language constraint graph representation learning for unsupervised vehicle re-identification

Existing vehicle re-identification methods rely on visual features to extract vehicle identity information. However, while individual visual features enable the model to learn limited semantic information, multimodal representations are difficult to extract. A vision-language constraint graph representation learning method guided by textual descriptions is proposed to exploit the cross-modal robustness capacity. Initially, the training set creates the unique conditional prompts for textual feature extraction. These prompts are employed to improve the understanding of visual modalities. We subsequently designed a vision-language constraint graph topology, where each training sample is considered a node in the graph. Under the dual constraints of visual and textual features, the relationship between graph nodes is further explored to construct more reliable positive and negative sample pairs for graph representation learning. Then, neighboring node label smoothing is introduced to mitigate label noise generated by visual feature clustering and achieved by combining pseudo-label assignment results from neighboring node pairs in the graph topology. Extensive experiments have confirmed that the proposed method achieves state-of-the-art performance by combining salient information from visual and textual modalities.

The contribution of prosody to machine classification of schizophrenia

We show how acoustic prosodic features, such as pitch and gaps, can be used computationally for detecting symptoms of schizophrenia from a single spoken response. We compare the individual contributions of acoustic and previously-employed text modalities to the algorithmic determination whether the speaker has schizophrenia. Our classification results clearly show that we can extract relevant acoustic features better than those textual ones. We find that, when combined with those acoustic features, textual features improve classification only slightly.

Seeing and Thinking about Urban Blue–Green Space: Monitoring Public Landscape Preferences Using Bimodal Data

Urban blue–green spaces (UBGSs) are a significant avenue for addressing the worldwide mental health crisis. To effectively optimise landscape design and management for the promotion of health benefits from UBGS, it is crucial to objectively understand public preferences. This paper proposes a method to evaluate public landscape preference from the perspective of seeing and thinking, takes the examples of seven parks around the Dianchi Lake in Kunming, China, and analyses the social media data by using natural language processing technology and image semantic segmentation technology. The conclusions are as follows: (1) The public exhibits significantly high positive sentiments towards various UBGSs, with over 93% of comments expressed positive sentiments. (2) Differences exist in the frequency and perception of landscape features between image and text modalities. Landscape elements related to stability are perceived more in images than in text, while dynamic and experiential elements are perceived more in text than in images. (3) In both modalities, the distinctive landscape features of parks are more frequently perceived and preferred by the public. In the end, the intrinsic links between landscape elements and public sentiment and preferences are discussed, and suggestions for design and management improvements are made to consolidate their health benefits to the public.

Enhancing Multimodal Emotion Recognition through Attention Mechanisms in BERT and CNN Architectures

Emotion detection holds significant importance in facilitating human–computer interaction, enhancing the depth of engagement. By integrating this capability, we pave the way for forthcoming AI technologies to possess a blend of cognitive and emotional understanding, bridging the divide between machine functionality and human emotional complexity. This progress has the potential to reshape how machines perceive and respond to human emotions, ushering in an era of empathetic and intuitive artificial systems. The primary research challenge involves developing models that can accurately interpret and analyze emotions from both auditory and textual data, whereby auditory data require optimizing CNNs to detect subtle and intense emotional fluctuations in speech, and textual data necessitate access to large, diverse datasets to effectively capture nuanced emotional cues in written language. This paper introduces a novel approach to multimodal emotion recognition, seamlessly integrating speech and text modalities to accurately infer emotional states. Employing CNNs, we meticulously analyze speech using Mel spectrograms, while a BERT-based model processes the textual component, leveraging its bidirectional layers to enable profound semantic comprehension. The outputs from both modalities are combined using an attention-based fusion mechanism that optimally weighs their contributions. The proposed method here undergoes meticulous testing on two distinct datasets: Carnegie Mellon University’s Multimodal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) dataset and the Multimodal Emotion Lines Dataset (MELD). The results demonstrate superior efficacy compared to existing frameworks, achieving an accuracy of 88.4% and an F1-score of 87.9% on the CMU-MOSEI dataset, and a notable weighted accuracy (WA) of 67.81% and a weighted F1 (WF1) score of 66.32% on the MELD dataset. This comprehensive system offers precise emotion detection and introduces several significant advancements in the field.

SEMScene: Semantic-Consistency Enhanced Multi-Level Scene Graph Matching for Image-Text Retrieval

Image-text retrieval, a fundamental cross-modal task, performs similarity reasoning for images and texts. The primary challenge for image-text retrieval is cross-modal semantic heterogeneity, where the semantic features of visual and textual modalities are rich but distinct. Scene graph is an effective representation for images and texts as it explicitly models objects and their relations. Existing scene graph based methods have not fully taken the features regarding various granularities implicit in scene graph into consideration (e.g. triplets), the inadequate feature matching incurs the absence of non-trivial semantic information (e.g. inner relations among triplets). Therefore, we propose a S emantic-Consistency E nhanced M ulti-Level Scene Graph Matching (SEMScene) network, which exploits the semantic relevance between visual and textual scene graphs from fine-grained to coarse-grained. Firstly, under the scene graph representation, we perform feature matching including low-level node matching, mid-level semantic triplet matching, and high-level holistic scene graph matching. Secondly, to enhance the semantic-consistency for object-fused triplets carrying key correlation information, we propose a dual-step constraint mechanism in mid-level matching. Thirdly, to guide the model to learn the semantic-consistency of matched image-text pairs, we devise effective loss functions for each stage of the dual-step constraint. Comprehensive experiments on Flickr30K and MS-COCO datasets demonstrate that SEMScene achieves state-of-the-art performances with significant improvements.

HKA: A Hierarchical Knowledge Alignment Framework for Multimodal Knowledge Graph Completion

Recent years have witnessed the successful application of knowledge graph techniques in structured data processing, while how to incorporate knowledge from visual and textual modalities into knowledge graphs has been given less attention. To better organize them, Multimodal Knowledge Graphs (MKGs), comprising the structural triplets of traditional Knowledge Graphs (KGs) together with entity-related multimodal data (e.g., images and texts), have been introduced consecutively. However, it is still a great challenge to explore MKGs due to their inherent incompleteness. Although most existing Multimodal Knowledge Graph Completion (MKGC) approaches can infer missing triplets based on available factual triplets and multimodal information, they almost ignore the modal conflicts and supervisory effect, failing to achieve a more comprehensive understanding of entities. To address these issues, we propose a novel H ierarchical K nowledge A lignment ( HKA ) framework for MKGC. Specifically, a macro-knowledge alignment module is proposed to capture global semantic relevance between modalities for dealing with modal conflicts in MKG. Furthermore, a micro-knowledge alignment module is also developed to reveal the local consistency information through inter- and intra-modality supervisory effect more effectively. By integrating different modal predictions, a final decision can be made. Experimental results on three benchmark MKGC tasks have demonstrated the effectiveness of the proposed HKA framework.

Similar modality completion-based multimodal sentiment analysis under uncertain missing modalities

Recently, uncertain missing modalities in multimodal sentiment analysis (MSA) brings a new challenge for sentiment analysis. However, existing research cannot accurately complete the missing modalities, and fail to explore the advantages of the text modality in MSA. For the above problems, this work develops a Similar Modality Completion based-MSA model under uncertain missing modalities (termed as SMCMSA). Firstly, we construct the full modalities samples database (FMSD) by screening out the full modality samples from the whole multimodal dataset, and then predicting and marking the sentiment labels of each modality of the samples with three pre-trained unimodal sentiment analysis model (PTUSA). Next, for completing the uncertain missing modalities, we propose a set of missing modalities completion strategies based on the similar modalities selected from FMSD. For the completed multimodal data, we first encode the text, video and audio modality using the encoder of transformer, then we fuse the representation of text into the representations of video and audio under the guidance of a pre-trained model, thereby improving the quality of video and audio. Finally, we conduct sentiment classification based on the representations of text, video and audio with the softmax function respectively, and get the final decision with the decision-level fusion method. Based on benchmark datasets CMU-MOSI and IEMOCAP, extensive experiments have been conducted to verify that our proposed model SMCMSA has better performance than that of the state-of-the-art baseline models.The codes of our model are available at https://github.com/Astro2Sun/SMCMSA.

Learning Commonsense-aware Moment-Text Alignment for Fast Video Temporal Grounding

Grounding temporal video segments described in natural language queries effectively and efficiently is a crucial capability needed in vision-and-language fields. In this paper, we deal with the fast video temporal grounding (FVTG) task, aiming at localizing the target segment with high speed and favorable accuracy. Most existing approaches adopt elaborately designed cross-modal interaction modules to improve the grounding performance, which suffer from the test-time bottleneck. Although several common space-based methods enjoy the high-speed merit during inference, they can hardly capture the comprehensive and explicit relations between visual and textual modalities. In this paper, to tackle the dilemma of speed-accuracy tradeoff, we propose a commonsense-aware cross-modal alignment network (C 2 AN), which incorporates commonsense-guided visual and text representations into a complementary common space for fast video temporal grounding. Specifically, the commonsense concepts are explored and exploited by extracting the structural semantic information from a language corpus. Then, a commonsense-aware interaction module is designed to obtain bridged visual and text features by utilizing the learned commonsense concepts. Finally, to maintain the original semantic information of textual queries, a cross-modal complementary common space is optimized to obtain matching scores for performing FVTG. Extensive results on two challenging benchmarks show that our C 2 AN method performs favorably against state-of-the-arts while running at high speed. Our code is available at https://github.com/ZiyueWu59/CCA.

Sketch2Prototype: rapid conceptual design exploration and prototyping with generative AI

AbstractSketch2Prototype is an AI-based framework that transforms a hand-drawn sketch into a diverse set of 2D images and 3D prototypes through sketch-to-text, text-to-image, and image-to-3D stages. This framework, shown across various sketches, rapidly generates text, image, and 3D modalities for enhanced early-stage design exploration. We show that using text as an intermediate modality outperforms direct sketch-to-3D baselines for generating diverse and manufacturable 3D models. We find limitations in current image-to-3D techniques, while noting the value of the text modality for user-feedback.

PROMISE: A pre-trained knowledge-infused multimodal representation learning framework for medication recommendation

Electronic Health Records (EHRs) significantly enhance clinical decision-making, particularly in safe and effective medication recommendation based on complex patient data. Current methods, while encoding each medical event individually with domain-specific knowledge, inadequately harness multi-source domain knowledge and neglect the interrelations among various medical codes, the influence of historical patient visits, and the relevance of similar patient trajectories. To address these limitations, we present PROMISE, a multimodal medication recommendation framework that progressively learns patient representations from specific health states to a comprehensive view. PROMISE integrates domain knowledge into modality-specific encoders to improve local and global patient representations, facilitating enhanced medication recommendations through the interaction of patient representations from various modalities. Specifically, within the code modality, PROMISE utilizes encoding of EHR hypergraphs to learn patient representations featuring structured information. Simultaneously, in the text modality, it acquires patient representations with semantic information by encoding clinical texts obtained from tables. Our framework surpasses state-of-the-art baselines with up to 2.06% and 1.97% improvements on key metrics within the MIMIC-III and IV datasets, respectively, confirming its effectiveness and superiority.

World futures through RT’s eyes: multimodal dataset and interdisciplinary methodology

There is a need to develop new interdisciplinary approaches suitable for a more complete analysis of multimodal data. Such approaches need to go beyond case studies and leverage technology to allow for statistically valid analysis of the data. Our study addresses this need by engaging with the research question of how humans communicate about the future for persuasive and manipulative purposes, and how they do this multimodally. It introduces a new methodology for computer-assisted multimodal analysis of video data. The study also introduces the resulting dataset, featuring annotations for speech (textual and acoustic modalities) and gesticulation and corporal behaviour (visual modality). To analyse and annotate the data and develop the methodology, the study engages with 23 26-min episodes of the show ‘SophieCo Visionaries’, broadcast by RT (formerly ‘Russia Today’).

Audio-textual multi-label demographic recognition of Arabic speakers using deep learning

Speaker demographic recognition and segmentation analytics play a key role in offering personalized experiences across different automated industries and businesses. This paper aims at developing a multi-label demographic recognition system for Arabic speakers from audio and associated textual modalities. The system can detect age groups, genders, and dialects, but it can be easily extended to incorporate more demographic traits. The proposed method is based on deep learning for feature learning and recognition. Representations of audio modality are learned through 3D spectrogram and AlexNet CNN-based architecture. An AraBERT transformer is employed for learning representations of the textual modality. Additionally, a method is provided for fusing audio and textual representations. The effectiveness of the proposed method is evaluated using the Saudi Audio Dataset for Arabic (SADA), which is a recently published database containing audio recordings of TV shows in different Arabic dialects. The experimental findings show that when using models with standalone modalities for multi-label demographic classification, textual modality using AraBERT performed better than the audio modality represented using 3D spectrogram along with AlexNet CNN-based architecture. Furthermore, when combining both modalities, audio and textual, significant improvement has been attained for all demographic traits.

Category of learning modality in English texts

In the literature of linguistics, modality is a semantic category that expresses the attitude of the speaker to the content of his expression, the target designation of the speech, the relationship of the content of the expression with reality. The category of modality is a linguistic universal and stands out as one of the main categories of language. This article examines the category of modality, one of the most popular and interesting areas in modern English grammar. Different types of modality are defined, examples and features of working of modal words in English texts are given, detailed explanation is given to the presented usage examples. The relevance of the research is related to the necessity of a comprehensive review of the category of modality from the point of view of the theoretical grammar of the modern English language, the importance of systematizing modal words, and determining their status in linguistics.

EmoStyle: Emotion-Aware Semantic Image Manipulation with Audio Guidance

With the flourishing development of generative models, image manipulation is receiving increasing attention. Rather than text modality, several elegant designs have delved into leveraging audio to manipulate images. However, existing methodologies mainly focus on image generation conditional on semantic alignment, ignoring the vivid affective information depicted in the audio. We propose an Emotion-aware StyleGAN Manipulator (EmoStyle), a framework where affective information from audio can be explicitly extracted and further utilized during image manipulation. Specifically, we first leverage the multi-modality model ImageBind for initial cross-modal retrieval between images and music, and select the music-related image for further manipulation. Simultaneously, by extracting sentiment polarity from the lyrics of the audio, we generate an emotionally rich auxiliary music branch to accentuate the affective information. We then leverage pre-trained encoders to encode audio and the audio-related image into the same embedding space. With the aligned embeddings, we manipulate the image via a direct latent optimization method. We conduct objective and subjective evaluations on the generated images, and our results show that our framework is capable of generating images with specified human emotions conveyed in the audio.

What do they “meme”? A metaphor-aware multi-modal multi-task framework for fine-grained meme understanding

Fine-grained meme understanding aims to explore and comprehend the meanings of memes from multiple perspectives by performing various tasks, such as sentiment analysis, intention detection, and offensiveness detection. Existing approaches primarily focus on simple multi-modality fusion and individual task analysis. However, there remain several limitations that need to be addressed: (1) the neglect of incongruous features within and across modalities, and (2) the lack of consideration for correlations among different tasks. To this end, we leverage metaphorical information as text modality and propose a Metaphor-aware Multi-modal Multi-task Framework (M3F) for fine-grained meme understanding. Specifically, we create inter-modality attention enlightened by the Transformer to capture inter-modality interaction between text and image. Moreover, intra-modality attention is applied to model the contradiction between the text and metaphorical information. To learn the implicit interaction among different tasks, we introduce a multi-interactive decoder that exploits gating networks to establish the relationship between various subtasks. Experimental results on the MET-Meme dataset show that the proposed framework outperforms the state-of-the-art baselines in fine-grained meme understanding.

Text-based person search via cross-modal alignment learning

Text-based person search aims to use text descriptions to search for corresponding person images. However, due to the obvious pattern differences in image and text modalities, it is still a challenging problem to align the two modalities. Most existing approaches only consider semantic alignment within a global context or partial parts, lacking consideration of how to match image and text in terms of differences in model information. Therefore, in this paper, we propose an efficient Modality-Aligned Person Search network (MAPS) to address this problem. First, we suppress image-specific information by image feature style normalization to achieve modality knowledge alignment and reduce information differences between text and image. Second, we design a multi-granularity modal feature fusion and optimization method to enrich the modal features. To address the problem of useless and redundant information in the multi-granularity fused features, we propose a Multi-granularity Feature Self-optimization Module (MFSM) to adaptively adjust the corresponding contributions of different granularities in the fused features of the two modalities. Finally, to address the problem of information inconsistency in the training and inference stages, we propose a Cross-instance Feature Alignment (CFA) to help the network enhance category-level generalization ability and improve retrieval performance. Extensive experiments demonstrate that our MAPS achieves state-of-the-art performance on all text-based person search datasets, and significantly outperforms other existing methods.

A Systematic Review of Big Data Driven Education Evaluation

The rapid development of artificial intelligence has driven the transformation of educational evaluation into big data-driven. This study used a systematic literature review method to analyzed 44 empirical research articles on the evaluation of big data education. Firstly, it has shown an increasing trend year by year, and is mainly published in thematic journals such as educational technology, science education, and language teaching. Chinese and American researchers have made the greatest contributions in this field. Secondly, the algorithmic models for big data education evaluation research are diverse, the text modality is the most popular, the evaluation subjects are mainly college students, with fewer primary and secondary school students, and science is the discipline that most commonly applies big data education evaluation. The evaluation objectives of big data education evaluation mainly focus on five aspects: high-order thinking analysis, learning performance prediction, learning emotion recognition, teaching management decision-making, and evaluation mode optimization, and the text modality is widely used for data collection in high-order thinking analysis; regardless of the evaluation objectives, higher education students are the most widely evaluated objects; the science discipline is the main field of using big data technology to empower teaching evaluation. Thirdly, the current research topics of big data education evaluation mainly focus on online learning behavior and environmental participation evaluation, process assessment of learning motivation and emotional analysis, development and optimization of subject domain big data models, cognitive diagnosis and high-order thinking skills evaluation, and design of learning analysis frameworks based on data mining.

How Does the High Court Interpret the Constitution? A Qualitative Analysis between 2019–21

Theorists of legal interpretation often argue that their theory describes or fits well with legal practice without using empirical evidence to support such claims. In this article, we provide a proof of concept for how such claims can be established using Critical Discourse Analysis – a qualitative method of coding texts – as applied to High Court decisions. Particularly, we assess whether a slightly modified version of Philip Bobbitt’s theory of constitutional modalities can be used to describe Australian constitutional interpretation working backwards from the start date of the project in August 2021. We find that Bobbitt’s modalities were used by High Court judges in the period studied. Predominantly, the High Court used the doctrinal modality supplemented strongly by textual and structural modalities. The ubiquitous use of doctrine to interpret the Australian Constitution in the period studied suggests a need for a greater understanding of doctrine as an interpretive modality.

Attr4Vis: Revisiting Importance of Attribute Classification in Vision-Language Models for Video Recognition

Vision-language models (VLMs), pretrained on expansive datasets containing image-text pairs, have exhibited remarkable transferability across a diverse spectrum of visual tasks. The leveraging of knowledge encoded within these potent VLMs holds significant promise for the advancement of effective video recognition models. A fundamental aspect of pretrained VLMs lies in their ability to establish a crucial bridge between the visual and textual domains. In our pioneering work, we introduce the Attr4Vis framework, dedicated to exploring knowledge transfer between Video and Text modalities to bolster video recognition performance. Central to our contributions is the comprehensive revisitation of Text-to-Video classifier initialization, a critical step that refines the initialization process and streamlines the integration of our framework, particularly within existing Vision-Language Models (VLMs). Furthermore, we emphasize the adoption of dense attribute generation techniques, shedding light on their paramount importance in video analysis. By effectively encoding attribute changes over time, these techniques significantly enhance event representation and recognition within videos. In addition, we introduce an innovative Attribute Enrichment Algorithm aimed at enriching set of attributes by large language models (LLMs) like ChatGPT. Through the seamless integration of these components, Attr4Vis attains a state-of-the-art accuracy of 91.5% on the challenging Kinetics-400 dataset using the InternVideo model.