Multimodal Context Research Articles

The Wildfire Dataset: Enhancing Deep Learning-Based Forest Fire Detection with a Diverse Evolving Open-Source Dataset Focused on Data Representativeness and a Novel Multi-Task Learning Approach

This study explores the potential of RGB image data for forest fire detection using deep learning models, evaluating their advantages and limitations, and discussing potential integration within a multi-modal data context. The research introduces a uniquely comprehensive wildfire dataset, capturing a broad array of environmental conditions, forest types, geographical regions, and confounding elements, aiming to reduce high false alarm rates in fire detection systems. To ensure integrity, only public domain images were included, and a detailed description of the dataset’s attributes, URL sources, and image resolutions is provided. The study also introduces a novel multi-task learning approach, integrating multi-class confounding elements within the framework. A pioneering strategy in the field of forest fire detection, this method aims to enhance the model’s discriminatory ability and decrease false positives. When tested against the wildfire dataset, the multi-task learning approach demonstrated significantly superior performance in key metrics and lower false alarm rates compared to traditional binary classification methods. This emphasizes the effectiveness of the proposed methodology and the potential to address confounding elements. Recognizing the need for practical solutions, the study stresses the importance of future work to increase the representativeness of training and testing datasets. The evolving and publicly available wildfire dataset is anticipated to inspire innovative solutions, marking a substantial contribution to the field.

Intact speech-gesture integration in narrative recall by adults with moderate-severe traumatic brain injury

PurposeReal-world communication is situated in rich multimodal contexts, containing speech and gesture. Speakers often convey unique information in gesture that is not present in the speech signal (e.g., saying “He searched for a new recipe” while making a typing gesture). We examine the narrative retellings of participants with and without moderate-severe traumatic brain injury across three timepoints over two online Zoom sessions to investigate whether people with TBI can integrate information from co-occurring speech and gesture and if information from gesture persists across delays. Methods60 participants with TBI and 60 non-injured peers watched videos of a narrator telling four short stories. On key details, the narrator produced complementary gestures that conveyed unique information. Participants retold the stories at three timepoints: immediately after, 20-min later, and one-week later. We examined the words participants used when retelling these key details, coding them as a Speech Match (e.g., “He searched for a new recipe”), a Gesture Match (e.g., “He searched for a new recipe online), or Other (“He looked for a new recipe”). We also examined whether participants produced representative gestures themselves when retelling these details. ResultsDespite recalling fewer story details, participants with TBI were as likely as non-injured peers to report information from gesture in their narrative retellings. All participants were more likely to report information from gesture and produce representative gestures themselves one-week later compared to immediately after hearing the story. ConclusionWe demonstrated that speech-gesture integration is intact after TBI in narrative retellings. This finding has exciting implications for the utility of gesture to support comprehension and memory after TBI and expands our understanding of naturalistic multimodal language processing in this population.

Balancing Project Schedule, Cost, and Value under Uncertainty: A Reinforcement Learning Approach

Industrial projects are plagued by uncertainties, often resulting in both time and cost overruns. This research introduces an innovative approach, employing Reinforcement Learning (RL), to address three distinct project management challenges within a setting of uncertain activity durations. The primary objective is to identify stable baseline schedules. The first challenge encompasses the multimode lean project management problem, wherein the goal is to maximize a project’s value function while adhering to both due date and budget chance constraints. The second challenge involves the chance-constrained critical chain buffer management problem in a multimode context. Here, the aim is to minimize the project delivery date while considering resource constraints and duration-chance constraints. The third challenge revolves around striking a balance between the project value and its net present value (NPV) within a resource-constrained multimode environment. To tackle these three challenges, we devised mathematical programming models, some of which were solved optimally. Additionally, we developed competitive RL-based algorithms and verified their performance against established benchmarks. Our RL algorithms consistently generated schedules that compared favorably with the benchmarks, leading to higher project values and NPVs and shorter schedules while staying within the stakeholders’ risk thresholds. The potential beneficiaries of this research are project managers and decision-makers who can use this approach to generate an efficient frontier of optimal project plans.

Multi-modal graph contrastive encoding for neural machine translation

As an important extension of conventional text-only neural machine translation (NMT), multi-modal neural machine translation (MNMT) aims to translate input source sentences paired with images into the target language. Although a lot of MNMT models have been proposed to perform multi-modal semantic fusion, they do not consider fine-grained semantic correspondences between semantic units of different modalities (i.e., words and visual objects), which can be exploited to refine multi-modal representation learning via fine-grained semantic interactions. To address this issue, we propose a graph-based multi-modal fusion encoder for NMT. Concretely, we first employ a unified multi-modal graph to represent the input sentence and image, in which the multi-modal semantic units are considered as the nodes in the graph, connected by two kinds of edges with different semantic relationships. Then, we stack multiple graph-based multi-modal fusion layers that iteratively conduct intra- and inter-modal interactions to learn node representations. Finally, via an attention mechanism, we induce a multi-modal context from the top node representations for the decoder. Particularly, we introduce a progressive contrastive learning strategy based on the multi-modal graph to refine the training of our proposed model, where hard negative samples are introduced gradually. To evaluate our model, we conduct experiments on commonly-used datasets. Experimental results and analysis show that our MNMT model obtains significant improvements over competitive baselines, achieving state-of-the-art performance on the Multi30K dataset.

Specific facial signals associate with categories of social actions conveyed through questions.

The early recognition of fundamental social actions, like questions, is crucial for understanding the speaker's intended message and planning a timely response in conversation. Questions themselves may express more than one social action category (e.g., an information request "What time is it?", an invitation "Will you come to my party?" or a criticism "Are you crazy?"). Although human language use occurs predominantly in a multimodal context, prior research on social actions has mainly focused on the verbal modality. This study breaks new ground by investigating how conversational facial signals may map onto the expression of different types of social actions conveyed through questions. The distribution, timing, and temporal organization of facial signals across social actions was analysed in a rich corpus of naturalistic, dyadic face-to-face Dutch conversations. These social actions were: Information Requests, Understanding Checks, Self-Directed questions, Stance or Sentiment questions, Other-Initiated Repairs, Active Participation questions, questions for Structuring, Initiating or Maintaining Conversation, and Plans and Actions questions. This is the first study to reveal differences in distribution and timing of facial signals across different types of social actions. The findings raise the possibility that facial signals may facilitate social action recognition during language processing in multimodal face-to-face interaction.

Incorporating multimodal context information into traffic speed forecasting through graph deep learning

Accurate traffic speed forecasting is a prerequisite for anticipating future traffic status and increasing the resilience of intelligent transportation systems. However, most studies ignore the involvement of context information ubiquitously distributed over the urban environment to boost speed prediction. The diversity and complexity of context information also hinder incorporating it into traffic forecasting. Therefore, this study proposes a multimodal context-based graph convolutional neural network (MCGCN) model to fuse context data into traffic speed prediction, including spatial and temporal contexts. The proposed model comprises three modules, ie (a) hierarchical spatial embedding to learn spatial representations by organizing spatial contexts from different dimensions, (b) multivariate temporal modeling to learn temporal representations by capturing dependencies of multivariate temporal contexts and (c) attention-based multimodal fusion to integrate traffic speed with the spatial and temporal context representations for multi-step speed prediction. We conduct extensive experiments in Singapore. Compared to the baseline model (spatial-temporal graph convolutional network, STGCN), our results demonstrate the importance of multimodal contexts with the mean-absolute-error improvement of 0.29 km/h, 0.45 km/h and 0.89 km/h in 30-min, 60-min and 120-min speed prediction, respectively. We also explore how different contexts affect traffic speed forecasting, providing references for stakeholders to understand the relationship between context information and transportation systems.

Intersensory redundancy impedes face recognition in 12-month-old infants.

This study examined the role of intersensory redundancy on 12-month-old infants' attention to and processing of face stimuli. Two experiments were conducted. In Experiment 1, 72 12-month-olds were tested using an online platform called Lookit. Infants were familiarized with two videos of an actor reciting a children's story presented simultaneously. A soundtrack either matched one of the videos (experimental condition) or neither of the videos (control condition). Visual-paired comparison (VPC) trials were completed to measure looking preferences for the faces presented synchronously and asynchronously during familiarization and for novel faces. Neither group displayed looking preferences during the VPC trials. It is possible that the complexity of the familiarization phase made the modality-specific face properties (i.e., facial characteristics and configuration) difficult to process. In Experiment 2, 56 12-month-old infants were familiarized with the video of only one actor presented either synchronously or asynchronously with the soundtrack. Following familiarization, participants completed a VPC procedure including the familiar face and a novel face. Results from Experiment 2 showed that infants in the synchronous condition paid more attention during familiarization than infants in the asynchronous condition. Infants in the asynchronous condition demonstrated recognition of the familiar face. These findings suggest that the competing face stimuli in the Experiment 1 were too complex for the facial characteristics to be processed. The procedure in Experiment 2 led to increased processing of the face in the asynchronous presentation. These results indicate that intersensory redundancy in the presentation of synchronous audiovisual faces is very salient, discouraging the processing of modality-specific visual properties. This research contributes to the understanding of face processing in multimodal contexts, which have been understudied, although a great deal of naturalistic face exposure occurs multimodally.

Automated segmentation of the human supraclavicular fat depot via deep neural network in water-fat separated magnetic resonance images.

Human brown adipose tissue (BAT), mostly located in the cervical/supraclavicular region, is a promising target in obesity treatment. Magnetic resonance imaging (MRI) allows for mapping the fat content quantitatively. However, due to the complex heterogeneous distribution of BAT, it has been difficult to establish a standardized segmentation routine based on magnetic resonance (MR) images. Here, we suggest using a multi-modal deep neural network to detect the supraclavicular fat pocket. A total of 50 healthy subjects [median age/body mass index (BMI) =36 years/24.3 kg/m2] underwent MRI scans of the neck region on a 3 T Ingenia scanner (Philips Healthcare, Best, Netherlands). Manual segmentations following fixed rules for anatomical borders were used as ground truth labels. A deep learning-based method (termed as BAT-Net) was proposed for the segmentation of BAT on MRI scans. It jointly leveraged two-dimensional (2D) and three-dimensional (3D) convolutional neural network (CNN) architectures to efficiently encode the multi-modal and 3D context information from multi-modal MRI scans of the supraclavicular region. We compared the performance of BAT-Net to that of 2D U-Net and 3D U-Net. For 2D U-Net, we analyzed the performance difference of implementing 2D U-Net in three different planes, denoted as 2D U-Net (axial), 2D U-Net (coronal), and 2D U-Net (sagittal). The proposed model achieved an average dice similarity coefficient (DSC) of 0.878 with a standard deviation of 0.020. The volume segmented by the network was smaller compared to the ground truth labels by 9.20 mL on average with a mean absolute increase in proton density fat fraction (PDFF) inside the segmented regions of 1.19 percentage points. The BAT-Net outperformed all implemented 2D U-Nets and the 3D U-Nets with average DSC enhancement ranging from 0.016 to 0.023. The current work integrates a deep neural network-based segmentation into the automated segmentation of supraclavicular fat depot for quantitative evaluation of BAT. Experiments show that the presented multi-modal method benefits from leveraging both 2D and 3D CNN architecture and outperforms the independent use of 2D or 3D networks. Deep learning-based segmentation methods show potential towards a fully automated segmentation of the supraclavicular fat depot.

Counterfactual Scenario-relevant Knowledge-enriched Multi-modal Emotion Reasoning

Multi-modal video emotion reasoning (MERV) has recently attracted increasing attention due to its potential application in human-computer interaction. This task needs to not only recognize utterance-level emotions for conspicuous speakers, but also perceive the emotions of non-speakers in videos. Existing methods focus on modeling multi-modal multi-level contexts to capture emotion-relevant clues from the complex scenarios in videos. However, the context information is far from enough to infer the emotion labels of non-speakers due to the large gap between the scenario situation and emotions labels. Inspired by the observation that humans can find solutions to complex problems with the leverage of experience and knowledge, we propose SK-MER , a Scenario-relevant Knowledge-enhanced Multi-modal Emotion Reasoning framework for MERV task, which can leverage external knowledge to enhance the video scenario understanding and emotion reasoning. Specifically, we use scenario concepts extracted from videos to build knowledge subgraphs from external knowledge bases. The knowledge subgraphs are then utilized to obtain scenario-relevant knowledge representations through dynamic knowledge graph attention. Next, we incorporate the knowledge representations into context modeling to enhance emotion reasoning with external scenario-relevant knowledge. In addition, we propose a counterfactual knowledge representation learning approach to obtain more effective scenario-relevant knowledge representations. Extensive experimental results on MEmoR dataset show that the proposed SK-MER framework achieves new state-of-the-art results.

Transformer-Based Visual Grounding with Cross-Modality Interaction

This article tackles the challenging yet important task of Visual Grounding (VG), which aims to localize a visual region in the given image referred by a natural language query. Existing efforts on the VG task are twofold: (1) two-stage methods first extract region proposals and then rank them according to their similarities with the referring expression, which usually leads to suboptimal results due to the quality of region proposals; (2) one-stage methods usually predict all the possible coordinates of the target region online by leveraging modern object detection architectures, which pay little attention to cross-modality correlations and have limited generalization ability. To better address the task, we present an effective transformer-based end-to-end visual grounding approach, which focuses on capturing the cross-modality correlations between the referring expression and visual regions for accurately reasoning the location of the target region. Specifically, our model consists of a feature encoder, a cross-modality interactor, and a modality-agnostic decoder. The feature encoder is employed to capture the intra-modality correlation, which models the linguistic context in query and the spatial dependency in image respectively. The cross-modality interactor endows the model with the capability of highlighting the localization-relevant visual and textual cues by mutual verification of vision and language, which plays a key role in our model. The decoder learns a consolidated token representation enriched by multi-modal contexts and further directly predicts the box coordinates. Extensive experiments on five public benchmark datasets with quantitative and qualitative analysis clearly demonstrate the effectiveness and rationale of our proposed method.

Multisite regulation integrates multimodal context in sensory circuits to control persistent behavioral states in C. elegans

Maintaining or shifting between behavioral states according to context is essential for animals to implement fitness-promoting strategies. How the integration of internal state, past experience and sensory inputs orchestrates persistent multidimensional behavioral changes remains poorly understood. Here, we show that C. elegans integrates environmental temperature and food availability over different timescales to engage in persistent dwelling, scanning, global or glocal search strategies matching thermoregulatory and feeding needs. Transition between states, in each case, involves regulating multiple processes including AFD or FLP tonic sensory neurons activity, neuropeptide expression and downstream circuit responsiveness. State-specific FLP-6 or FLP-5 neuropeptide signaling acts on a distributed set of inhibitory GPCR(s) to promote scanning or glocal search, respectively, bypassing dopamine and glutamate-dependent behavioral state control. Integration of multimodal context via multisite regulation in sensory circuits might represent a conserved regulatory logic for a flexible prioritization on the valence of multiple inputs when operating persistent behavioral state transitions.

Effects of Bilingual Videos in National Innovation Project in College English Listening Teaching Under the Influences of Multi-modal Theory

Multi-modal discourse refers to the phenomenon of using multiple senses such as auditory, visual, and tactile senses to communicate through multiple means and symbolic resources such as language, images, sounds, and actions. This paper reviews the theory of multi-modal discourse and studies the application of multi-modal discourse in English teaching at home and abroad, and conducts an empirical study of a new university English listening teaching mode in a multi-modal context by using comparative analysis method, taking Y University as an example. Secondly, through comparative analysis, one control group and four experimental groups were set up , the relevant materials of this experiment are all from the impression Yue platform, and video, audio, Chinese subtitles and English subtitles were used as variables, and through analyzing the experimental results, it was found that the reasonable use of multi-modal combinations can significantly improve students’ listening comprehension, and the use of audio-visual combinations and English subtitles is more beneficial to students’ listening The experimental results showed that the use of multi-modal combinations can significantly improve students’ listening comprehension, and the use of audio-visual combinations and English subtitles is more beneficial to students' listening comprehension.

Conceptual complexes in multimodal advertising

Several scholars have dealt with the workings of metaphor and metonymy in multimodal advertisements (see Forceville, 2009; Uriós-Aparisi, 2009; or Pérez-Sobrino, 2017, among others). The present study investigates conceptual complexity to broaden the set of analytical categories to be used in multimodal analysis by making use of some of the latest developments on conceptual complexes, or principled combinations of cognitive models (e.g., frames, metaphors, metonymies), as discussed in Ruiz de Mendoza (2017, 2021) to a multimodal context. Work on conceptual complexity in Cognitive Linguistics has taken two main directions. One is provided by Blending Theory, which focuses on accounting for the emergence of new structure not present in the contributing conceptual constructs after selected integration. Another direction studies patterns of conceptual interaction with a view to finding regularities that can be formulated as high-level generalizations. We will adopt this second direction. The main aim of the study is to test these analytical categories and principles of knowledge organization in terms of their communicative impact within a multimodal environment. A subsidiary aim is to further develop the theoretical apparatus underlying this initial work. Analysing a corpus of 62 multimodal advertisements, we found that: (i) the nature of a frame determines its function, i.e., matrix frames are receiving frames, which ‘situationalize’ conceptual structure, whereas donor frames play a focal role; (ii) sometimes there is no frame integration but rather internal development within a given frame, which is possible thanks to the incorporation of an external element that is not integrated, but simply facilitates the development of the frame; (iii) there are also cases in which there is frame composition instead of integration; (iv) metonymy proves to be a licensing factor previous to integration, and (v) high-level non-metaphorical correlations can act as cues for the activation of metaphorical frameworks.

I See What You're Hearing: Facilitating The Effect of Environment on Perceived Emotion While Teleconferencing

Our perception of emotion is highly contextual. Changes in the environment can affect our narrative framing, and thus augment our emotional perception of interlocutors. User environments are typically heavily suppressed due to the technical limitations of commercial videoconferencing platforms. As a result, there is often a lack of contextual awareness while participating in a video call, and this affects how we perceive the emotions of conversants. We present a videoconferencing module that visualizes the user's aural environment to enhance awareness between interlocutors. The system visualizes environmental sound based on its semantic and acoustic properties. We found that our visualization system was about 50% effective at eliciting emotional perceptions in users that was similar to the response elicited by environmental sound it replaced.The contributed system provides a unique approach to facilitate ambient awareness on an implicit emotional level in situations where multimodal environmental context is suppressed.

BAFN: Bi-Direction Attention Based Fusion Network for Multimodal Sentiment Analysis

Attention-based networks currently identify their effectiveness in multimodal sentiment analysis. However, existing methods ignore the redundancy of auxiliary modalities. More importantly, existing methods only attend to top-down attention (static process) or down-top attention (implicit process), leading to the coarse-grained multimodal sentiment context. In this paper, during the preprocessing period, we first propose the multimodal dynamic enhanced block to capture the intra-modality sentiment context. This can effectively decrease the intra-modality redundancy of auxiliary modalities. Furthermore, the bi-direction attention block is proposed to capture fine-grained multimodal sentiment context via the novel bi-direction multimodal dynamic routing mechanism. Specifically, the bi-direction attention block first highlights the explicit and low-level multimodal sentiment context. Then, the low-level multimodal context is transmitted to a carefully designed bi-direction multimodal dynamic routing procedure. This allows us to dynamically update and investigate high-level and much more fine-grained multimodal sentiment contexts. The experiments demonstrate that our fusion network can achieve state-of-the-art performance. Notably, our model outperforms the best baseline on the metric ‘Acc-7’ with an improvement of 6.9%.

Multimodal Named Entity Recognition with Bottleneck Fusion and Contrastive Learning

Multimodal named entity recognition (MNER) is the task of recognizing named entities in multimodal context. Existing methods focus on utilizing co-attention mechanism to discover the relationships between multiple modalities. However, they still have two deficiencies: First, current methods fail to fuse the multimodal representations in a fine-grained way, which may bring noise of visual modalities. Second, current methods ignore bridging the semantic gap between heterogeneous modalities. To solve the above issues, we propose a novel MNER method with bottleneck fusion and contrastive learning (BFCL). Specifically, we first incorporate the transformer-based bottleneck fusion mechanism, subsequently, information between different modalities can only be exchanged through several bottleneck tokens, thus reducing the noise propagation. Then we propose two decoupled image-text contrastive losses to align the unimodal representations, making the representations of semantically similar modalities closer, while the representations of semantically different modalities farther away. Experimental results demonstrate that our method is competitive to the state-of-the-art models, and achieves 74.54% and 85.70% F1-scores on Twitter-2015 and Twitter-2017 datasets, respectively.

Global and local priming in a multi-modal context.

Perceptual information can be processed at many different scales, from featural details to entire scenes. Attentional selection of different scales has been studied using hierarchical stimuli, with research elucidating a variety of biases in local and global attentional selection (due to, e.g., stimulus properties, brain injury, and experience). In this study, the emphasis is on biases produced through recent experience, or level-specific priming effects, which have been demonstrated within both the visual and auditory modalities. Namely, when individuals attend to local information, they are subsequently biased to attend locally (and similarly so with global attention). Here, these level-specific priming effects are investigated in a multi-modal context to determine whether cross-modal interactions occur between visual and auditory modalities during hierarchical processing. Specifically, the study addresses if attentional selection of local or global information in the visual modality subsequently biases auditory attentional selection to that level, and vice versa (i.e., level-priming). Though expected identity priming effects emerged in the study, no cross-modal level-priming effects manifested. Furthermore, the multi-modal context eliminated the well-established within-modality level-specific priming effects. Thus, though the study does reveal a multi-modal effect, it was not a level-based effect. Instead, paradoxically, the multi-modal context eliminated attentional scope biases (i.e., level-priming) within uni-modal transitions. In other words, when visual and auditory information are equally likely require attention, no persistence emerges for processing local or global information over time, even within a single modality.

Dynamic dual graph networks for textbook question answering

Textbook Question Answering (TQA) is a complex task oriented to multi-modal context, which requires reasoning on a diagram and a long essay to get the correct answer. There are mainly two related issues for the task. First, diagrams are mostly abstract expressions of real world and some constituents with similar appearance may have different semantics, which makes it difficult to understand them effectively. Secondly, a long essay contains abundant and useful information for question answering, which shows that it is vital to extract the relevant information from the abundant text and then perform reasoning on it. To address the two issues, we propose a new model, Dynamic Dual Graph Networks (DDGNet), which performs question-guided multi-step reasoning on the dynamic directed Diagram Graph Network (DGN) for the diagram and Textual Graph Network (TGN) for the most related paragraph extracted from a long essay. Specifically, DGN combines text features with positional features of text boxes in the diagram as the node feature to avoid the ambiguity of visual features for the abstract constituents and help express explicit semantics. TGN uses the representation of each sentence in the most related paragraph as the node feature to learn the contextualized interaction of the useful information in the graph reasoning process. For the reasoning strategy, we propose a question-guided multi-step graph reasoning method to update both DGN and TGN dynamically under the question guidance in every step. Experimental results show that our proposed model outperforms the baselines on the TQA dataset. Moreover, extensive ablation studies are also conducted to analyze the effectiveness of our proposed model.

New Directions in Cognitive Grammar and Style

New Directions in Cognitive Grammar and Style

Music and discourse: A systemic-functional approach for music analysis in multimodal contexts

Meaning in music constitutes an issue that have risen (and still rises) a strong debate. Although it has been approached from different perspectives there seems to be no consensus for its theorization. From a Social Semiotic perspective, music can be analysed as a part of a bigger construction, sharing the task of making meaning with other modes (verbal, visual, etc.). This paper aims to make a contribution to multimodal studies proposing a way to include music analysis in Multimodal Critical Discourse Analysis. We adopt as framework Halliday’s theory (1978 ; 1994) , although we start from Kress and van Leeuwen (2020) adaptations and include McKerrell (2015) and van Leeuwen (1999). Our working hypothesis is that, in multimodal contexts, music contributes with the construction of meaning, particularly ideational meaning, establishing a relation with other modes.