Multimodal Interaction Research Articles

Analysing educational dialogue around shared artefacts in technology-mediated contexts: a new coding framework

ABSTRACT Multimodal interactions that incorporate technology use can support rich new forms of educational dialogue that highlight differences between participants’ perspectives and make reasoning processes more explicit. Learning with technology is mediated by digitally represented knowledge artefacts that participants construct together and manipulate through dialogue. However, there is a need for research into the methods by which researchers and practitioners can understand the learning taking place in such complex environments where verbal and nonverbal interactions with technology are inter-connected and entangled with activities and material learning resources. This paper addresses that need. It offers a critical discussion of the underlying theory and presents a new 13-category coding scheme, ‘Tech-SEDA’. This is the first public version of a systematic, research-informed and practical tool for analysing verbal and nonverbal dialogic communication at the micro- or meso- level across diverse educational settings. Application is illustrated with examples of dialogic interaction across diverse technology contexts by learners located in classrooms or remotely – including online discussions, 3-dimensional modelling in augmented reality, software apps on tablets, Knowledge Forum, Scratch programming, interactive whiteboards, and more. The paper considers the methodological issues arising in capturing dynamic multimodal dialogue and in identifying the learning taking place.

A Multilayer Architecture towards the Development and Distribution of Multimodal Interface Applications on the Edge.

Today, Smart Assistants (SAs) are supported by significantly improved Natural Language Processing (NLP) and Natural Language Understanding (NLU) engines as well as AI-enabled decision support, enabling efficient information communication, easy appliance/device control, and seamless access to entertainment services, among others. In fact, an increasing number of modern households are being equipped with SAs, which promise to enhance user experience in the context of smart environments through verbal interaction. Currently, the market in SAs is dominated by products manufactured by technology giants that provide well designed off-the-shelf solutions. However, their simple setup and ease of use come with trade-offs, as these SAs abide by proprietary and/or closed-source architectures and offer limited functionality. Their enforced vendor lock-in does not provide (power) users with the ability to build custom conversational applications through their SAs. On the other hand, employing an open-source approach for building and deploying an SA (which comes with a significant overhead) necessitates expertise in multiple domains and fluency in the multimodal technologies used to build the envisioned applications. In this context, this paper proposes a methodology for developing and deploying conversational applications on the edge on top of an open-source software and hardware infrastructure via a multilayer architecture that simplifies low-level complexity and reduces learning overhead. The proposed approach facilitates the rapid development of applications by third-party developers, thereby enabling the establishment of a marketplace of customized applications aimed at the smart assisted living domain, among others. The supporting framework supports application developers, device owners, and ecosystem administrators in building, testing, uploading, and deploying applications, remotely controlling devices, and monitoring device performance. A demonstration of this methodology is presented and discussed focusing on health and assisted living applications for the elderly.

Enhancing Reflective and Conversational User Engagement in Argumentative Dialogues with Virtual Agents

In their process of information seeking, human users tend to selectively ignore information that contradicts their pre-existing beliefs or opinions. These so-called “self-imposed filter bubbles” (SFBs) pose a significant challenge for argumentative conversational agents aiming to facilitate critical, unbiased opinion formation on controversial topics. With the ultimate goal of developing a system that helps users break their self-imposed filter bubbles (SFBs), this paper aims to investigate the role of co-speech gestures, specifically examining how these gestures significantly contribute to achieving this objective. This paper extends current research by examining methods to engage users in cooperative discussions with a virtual human-like agent, encouraging a deep reflection on arguments to disrupt SFBs. Specifically, we investigate the agent’s non-verbal behavior in the form of co-speech gestures. We analyze whether co-speech gestures, depending on the conveyed information, enhance motivation, and thus conversational user engagement, thereby encouraging users to consider information that could potentially disrupt their SFBs. The findings of a laboratory study with 56 participants highlight the importance of non-verbal agent behaviors, such as co-speech gestures, in improving users’ perceptions of the interaction and the conveyed content. This effect is particularly notable when the content aims to challenge the user’s SFB. Therefore, this research offers valuable insights into enhancing user engagement in the design of multimodal interactions with future cooperative argumentative virtual agents.

Exploring the multimodal affordances of digital coding devices in fostering creative thinking in early childhood education

Digital technologies are increasingly acknowledged for their potential to enrich early childhood learning experiences, particularly in the area of creativity. However, research has tended to focus upon end products and skill development, and little is known about creative development processes. This study sought to explore young children's creative processes as they engaged with three digital coding devices over an eight-week period as part of a Digital Discovery Program. It employed a microethnography to provide nuanced observations of five young children's (aged 4–5 years) interactions and behaviours. Data was collected through approximately 40 hours of video recordings, capturing the dynamic and multimodal aspects of the childrenʼs play and learning experiences. A multimodal interaction analysis was used to examine the multimodal design affordances of each device as mapped against Murcia et al.'s (2020) A-E of Creativity Framework. The findings revealed a clear interplay between digital technologies and various modes of communication, with children demonstrating creativity through their verbal expressions, manipulation of the devices, and spatial understanding of their physical environment. By understanding the multimodal affordances of digital devices aligned with educational frameworks and embracing child-specific definitions of creativity, educators, curriculum designers, policymakers, and researchers can collectively promote creativity and enrich digital learning and play for young children.

Consistency and variability in multimodal parent-child social interaction: An at-home study using head-mounted eye trackers.

Real-time attention coordination in parent-toddler dyads is often studied in tightly controlled laboratory settings. These studies have demonstrated the importance of joint attention in scaffolding the development of attention and the types of dyadic behaviors that support early language learning. Little is known about how often these behaviors occur in toddlers' everyday lives. We brought wireless head-mounted eye trackers to families' homes to study the moment-to-moment patterns of toddlers' and parents' visual attention and manual activity in daily routines. Our sample consisted of English- and Spanish-speaking families who all reported being middle- or upper middle-class. Toddlers were 2 to 3 years old. Consistent with the findings from previous laboratory studies, we found variability in how frequently toddlers attended to named objects in two everyday activities-Object Play and Mealtime. We then tested whether parent-toddler joint attention in the seconds before a naming utterance increased toddler's attention on the named object. We found that joint attention accompanied by the attended object being held increased the child's attention to the labeled object during naming. We posit that in the rich, noisy world of toddlers' everyday lives, embodied attention plays a critical role in coordinating dyadic behaviors and creating informative naming moments. Our findings highlight the importance of studying toddlers' natural behavior in the real world. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Multimodal interactions in Stomoxys navigation reveal synergy between olfaction and vision

Stomoxys flies exhibit an attraction toward objects that offer no rewards, such as traps and targets devoid of blood or nectar incentives. This behavior provides an opportunity to develop effective tools for vector control and monitoring. However, for these systems to be sustainable and eco-friendly, the visual cues used must be specific to target vector(s). In this study, we modified the existing blue Vavoua trap, which was originally designed to attract biting flies, to create a deceptive host attraction system specifically biased toward attracting Stomoxys. Our research revealed that Stomoxys flies are attracted to various colors, with red proving to be the most attractive and selective color for Stomoxys compared to the other colors tested. Interestingly, our investigation of the cattle–Stomoxys interaction demonstrated that Stomoxys flies do not prefer a specific livestock fur color phenotype, despite variation in the spectrum. To create a realistic sensory impression of the trap in the Stomoxys nervous system, we incorporated olfactory cues from livestock host odors that significantly increased trap catches. The optimized novel polymer bead dispenser is capable of effectively releasing the attractive odor carvone + p-cresol, with strong plume strands and longevity. Overall, red trap baited with polymer bead dispenser is environmentally preferred.

AI in ambulatory care: Stakeholder needs, challenges and anticipations towards a hybrid, multimodal interaction AI-system

AI in ambulatory care: Stakeholder needs, challenges and anticipations towards a hybrid, multimodal interaction AI-system

Conversational AI and Chatbots: Enhancing User Experience on Websites

This research paper explores the transformative potential of conversational AI and chatbots in enhancing website user experience (UX). It addresses two key research questions: How do these technologies improve user engagement and satisfaction on websites, and what are the primary challenges in implementing them, along with effective solutions. The study examines case studies across diverse industries, including e-commerce, travel, healthcare, and finance, to gain insights into the underlying technologies powering conversational AI and chatbots, such as natural language processing (NLP), natural language understanding (NLU), and machine learning techniques. The paper highlights the significant benefits of integrating conversational AI and chatbots into websites, including providing personalized assistance, streamlining complex processes, ensuring 24/7 availability, and enhancing accessibility for users. However, the study also addresses the key challenges faced in implementation, ranging from handling ambiguity and context in natural language processing to ensuring data privacy and security, managing user expectations, and the need for continuous improvement and training. The research proposes solutions to these challenges, such as employing advanced NLP algorithms, robust API management tools, and establishing user feedback loops. Ethical considerations, including data privacy and addressing biases in AI responses, are also explored, emphasizing the importance of robust encryption, adherence to data privacy regulations, and advanced access control mechanisms. The paper concludes by providing a comprehensive overview of the current state and future directions of conversational AI and chatbots in enhancing website user experience, exploring emerging trends such as multimodal interactions, contextual awareness and personalization, integration with IoT devices, and the development of emotional intelligence and empathy in chatbots.

MMCL-CPI: A multi-modal compound-protein interaction prediction model incorporating contrastive learning pre-training

MotivationCompound-protein interaction (CPI) prediction plays a crucial role in drug discovery and drug repositioning. Early researchers relied on time-consuming and labor-intensive wet laboratory experiments. However, the advent of deep learning has significantly accelerated this progress. Most existing deep learning methods utilize deep neural networks to extract compound features from sequences and graphs, either separately or in combination. Our team’s previous research has demonstrated that compound images contain valuable information that can be leveraged for CPI task. However, there is a scarcity of multimodal methods that effectively combine sequence and image representations of compounds in CPI. Currently, the use of text-image pairs for contrastive language-image pre-training is a popular approach in the multimodal field. Further research is needed to explore how the integration of sequence and image representations can enhance the accuracy of CPI task. ResultsThis paper presents a novel method called MMCL-CPI, which encompasses two key highlights: 1) Firstly, we propose extracting compound features from two modalities: one-dimensional SMILES and two-dimensional images. This approach enables us to capture both sequence and spatial features, enhancing the prediction accuracy for CPI. Based on this, we design a novel multimodal model. 2) Secondly, we introduce a multimodal pre-training strategy that leverages comparative learning on a large-scale unlabeled dataset to establish the correspondence between SMILES string and compound’s image. This pre-training approach significantly improves compound feature representations for downstream CPI task. Our method has shown competitive results on multiple datasets.

DOE-based process optimization for development of efficient methods for purification of recombinant hepatitis B surface antigen from Pichia pastoris feedstock using Capto adhere resin

BackgroundThe multimodal chromatography resins, such as Capto adhere, are considered good candidates to be utilized in downstream processing due to their high capacity and selectivity; however, their multimodal interactions lead to an intricacy in the adsorption-desorption patterns and systematic characterization of conditions for process steps is necessary. MethodsCapto adhere, a strong ion exchanger with multimodal functionality, was used in this study for the final aim of recombinant hepatitis B surface antigen (rHBsAg) purification from Pichia pastoris (P. pastoris) industrial feedstock. Optimization of various parameters was done using the design of experiments (DOE) approach to determine the best binding and non-binding conditions. ResultsMaximum rHBsAg binding on Capto adhere occurred in 20 mM sodium acetate, pH 4.5, and a binding capacity of about 0.75 mg/ml was achieved, which was much higher than rHBsAg binding capacity of other resins reported so far. In elution optimization investigations, it was revealed that 1 M arginine (buffered in 50 mM sodium phosphate, pH 6.5) was the most efficient eluting agent. The binding and elution optimal conditions were utilized for further purification of rHBsAg from P. pastoris industrial feedstock in bind-elute mode, and the recovery and purity of the obtained rHBsAg were about 60% and 100%, respectively. Following optimization in the flow-through purification mode, the target protein recovery was significantly increased (up to 97%) and the target protein purity of more than 95% was achievable. SEC-HPLC analysis showed that the obtained retention times for the purified rHBsAg were similar to those reported previously. ConclusionsThese results suggest that Capto adhere under such optimized conditions can be considered as a good candidate for efficient purification of rHBsAg from P. pastoris industrial feedstock in downstream processing.

Improving visual grounding with multi-modal interaction and auto-regressive vertex generation

We propose a concise and consistent network focusing on multi-task learning of Referring Expression Comprehension (REC) and Segmentation (RES) within Visual grounding (VG). To simplify the model architecture and achieve parameter sharing, we reconstruct the Visual grounding task as a floating-point coordinate generation problem based on both image and text inputs. Consequently, rather than separately predicting bounding boxes and pixel-level segmentation masks, we represent them uniformly as a sequence of coordinate tokens and output two corner points of bounding boxes and polygon vertices autoregressively. To improve the accuracy of point generation, we introduce a regression-based decoder. Inspired by bilinear interpolation, this decoder can directly predict precise floating-point coordinates, thus avoiding quantization errors. Additionally, we devise a Multi-Modal Interaction Fusion (M2IF) to address the imbalance between visual and language features in the model. This module focuses visual information on regions relevant to textual descriptions while suppressing the influence of irrelevant areas. Based on our model, Visual grounding is realized through a unified network structure. Experiments conducted on five benchmark datasets (RefCOCO, RefCOCO+, RefCOCOg, ReferItGame and Flickr30K Entities) demonstrate that the proposed unified network outperforms or is on par with many existing task-customized models. Codes are available at https://github.com/LFUSST/MMI-VG.

Cross-cultural Communication Strategies of Forbidden City Cultural Documentaries on Overseas Social Media--Taking "I Repair Cultural Relics in the Forbidden City" on YouTube platform as an example

Telling a good Chinese story and enhancing the dissemination, cohesion, and influence of Chinese culture is an important direction to show China's cultural soft power, and an important issue to build a strong cultural country. Utilizing documentaries to creatively transform traditional Chinese culture and incorporate it into the dissemination of Chinese stories and Chinese discourse is conducive to the realization of "high-quality going out" of Chinese culture. This paper selects the documentary "I Repair Cultural Relics in the Forbidden City" released by CCTV on the overseas video platform YouTube as the research object and analyzes the cross-cultural communication strategy of this documentary on YouTube based on the SPEAKING theory under the method of communicative ethnography. It is found that the following points are usually more important to realize the success of Chinese culture "borrowing ship to go to the sea" using documentary communication: firstly, documentaries need to make use of overseas media platforms to carry out multimodal cross-interaction, turn abstract concepts of Chinese culture into visualization, and promote audiences' understanding and recognition of Chinese culture utilizing colorful expressions; secondly, to build a cultural network to promote the understanding and recognition of Chinese culture; and secondly, to build a cultural network to promote the understanding and recognition of Chinese culture using multimodal interaction. First, documentaries need to make use of overseas media platforms to engage in multimodal cross-interaction, turn abstract concepts of Chinese culture into figurative ones, and promote audiences' understanding of Chinese culture colorfully; second, they need to build a space of common cultural meanings, and use commonplace contents such as human fires to trigger the resonance of the overseas audience; and third, they need to merge the dialectical thinking of globalization and localization, and tell the story of China in international language. The in-depth study of documentaries on overseas video platforms in this paper is conducive to providing experience and reference for the production of future documentaries and has certain reference value for documentaries to better present a comprehensive and authentic Chinese culture to the world.

Unifying multimodal interactions for rumor diffusion prediction with global hypergraph modeling

A central issue in rumor surveillance and management is decoding the complex dynamics of rumor propagation, with an emphasis on predicting diffusion cascades. Recent studies focus on node embedding or the sequence of dissemination in a rumor cascade based on user social interaction, while neglecting interactions between users and rumors, as well as between various rumors. Online rumor diffusion is a complex system that encompasses two fundamental components: rumors and users. A comprehensive understanding of this system’s dynamics requires a global perspective. Consequently, it is necessary to develop models that capture the inherent multimodal interactions in rumor propagation. To tackle this challenge, we propose an HG2RLink framework of rumor diffusion prediction, which unifies multimodal interactions by global encoding. Specifically, our methodology begins with the establishment of a hypergraph structure. We then refine these interactions by leveraging a hypergraph neural network that aggregates users’ preferences for rumors. Two high-order graphs are generated to capture latent spatial interactions of rumors. By employing a learning approach for multimodal interactions, each rumor diffusion sequence is modeled with a long-range vision field of users in a deep neural network. Moreover, to validate the effectiveness of our method, we introduce a new dataset available for further exploration of new methods. Finally, experimental results show that HG2RLink outperforms other methods with improvements ranging from 0.8% to 7.6% for the Hits@k metric and from 0.4% to 2.6% for the MAP@k metric across four diverse datasets.

Joint training strategy of unimodal and multimodal for multimodal sentiment analysis

With the explosive growth of social media video content, research on multimodal sentiment analysis (MSA) has attracted considerable attention recently. Despite significant progress in MSA, there remains challenges: current research mostly focuses on learning either unimodal features or aspects of multimodal interactions, neglecting the importance of simultaneously considering both unimodal features and intermodal interactions. To address the aforementioned challenges, this paper proposes a fusion strategy called Joint Training of Unimodal and Multimodal (JTUM). Specifically, this strategy combines unimodal label generation module with cross-modal transformer. The unimodal label generation module aims to generate more distinctive labels for each unimodal input, facilitating more effective learning of unimodal representations. Meanwhile, cross-modal transformer is designed to treat each modality as a target modality and optimize it using other modalities as source modalities, thereby learning the interactions between each pair of modalities. By jointly training unimodal and multimodal tasks, our model can focus on individual modality features while learning the interactions between modalities. Finally, to better capture temporal information and make predictions, we also added self-attention transformer as sequence models. Experimental results on the CMU-MOSI and CMU-MOSEI datasets demonstrate that JTUM outperforms current main methods.

Transferable dual multi-granularity semantic excavating for partially relevant video retrieval

Partially Relevant Video Retrieval (PRVR) aims to retrieve partially relevant videos from many unlabeled and untrimmed videos according to the query, which is defined as the multiple instance learning problem. The challenge of PRVR is that it utilizes untrimmed videos, which are much closer to reality. The existing methods excavate video-text semantic consistency information insufficiently and lack the capacity to highlight the semantics of key representations. To tackle these issues, we propose a transferable dual multi-granularity semantic excavating network, called T-D3N, to focus on enhancing the learning of dual-modal representations. Specifically, we first introduce a novel transferable textual semantic learning strategy by designing Adaptive Multi-scale Semantic Mining (AMSM) component to excavate significant textual semantic from multiple perspectives. Second, T-D3N distinguishes the feature differences from the frame-wise perspective to better perform contrastive learning between positive and negative samples in the video feature domain, which can further distance the positive and negative samples and improve the probability of positive samples being retrieved by query. Finally, our model constructs multi-grained video temporal dependencies and conducts cross-grained core feature perception, which enables more sufficient multimodal interactions. Extensive experiments are performed on three benchmarks, i.e., ActivityNet Captions, Charades-STA, and TVR, our T-D3N achieves state-of-the-art results. Furthermore, we also confirm that our model is transferable on a broad range of multimodal tasks such as T2VR, VMR, and MMSum.

The Interactional Histories of Performance Bodies: From Describing to Depicting Proposed Ideas at Opera Rehearsals

ABSTRACT This article investigates opera performers’ ideas on what they can do on stage to music in the service of portraying the characters of an opera production. The performers both describe and depict their ideas, that is, tell and show them, in proposals. Using multimodal interaction analysis, the article examines how the relative deployment of depictions and descriptions in performer proposals change over time. During joint decision-making micro-histories, the proposals become increasingly depictive, as displayed access, alignment, and agreement with the ideas increase. If ideas originate in previous instructions and decisions, however, proposals are immediately constructed with depictions, and if ideas are resisted, they are never fully depicted. The article thus reveals the dialogical and interactional development of ideas on their way to materialize as artwork. The data consist of 20 hours of video recorded opera rehearsals in Swedish, English, and some Italian, which is the language of the libretto.

Ingredient-guided multi-modal interaction and refinement network for RGB-D food nutrition assessment

RGB-D food nutrition assessment entails the direct prediction of nutritional content in pairs of RGB and depth food images using signal processing technology. However, existing methods present numerous challenges related to accuracy and computational complexity. In this paper, we introduce the Ingredient-guided Multi-modal Interaction and Refinement Network (IMIR-Net), a novel framework for RGB-D food nutrition assessment. We first utilize a visual grounding model to accurately identify the dish region containing food in the input image with complex semantics. The identified RGB-D dish regions are processed through a multi-modal interaction and refinement module to compute the fused embedding. Concurrently, we introduce an ingredient guidance module to effectively capture ingredient textual information, thereby facilitating the acquisition of the enhanced fused embedding. Finally, we design a decoupled nutrition predictor to predict nutrition from the fused RGB-D embedding. The IMIR-Net can capture the rich multi-modal semantics for RGB-D food nutrition assessment and achieves state-of-the-art performances on the Nutrition5k dataset. Specifically, the percentage of mean absolute error (PMAE) of Calories, Mass, Fat, Carb and Protein reached 14.5%, 10.4%, 21.8%, 20.4% and 20.0%, respectively. Compared with the existing best method, our IMIR-Net demonstrated superior prediction accuracy, with a mean PMAE value of 17.4%, an improvement of 1.1%. In terms of computational complexity, the floating-point operations (FLOPs) and parameters reached 63.87 and 105.11 via our method, which is improved by 5.2% and 7.4%, respectively. The codes, data, and model of our method are publicly available at https://github.com/nianfd/IMIR-Net-DSP.

TRANSITIONAL AREAS OF DIGITAL GAME ART

Digital games, which constitute a part of new media studies, offer different reality experiences and new expressive environments to a player with multimodal interactions and hybrid reality practices offered by game engines. The alternative environments offered by digital game art create a kind of ‘transitional space’ in the participant’s experience process with ambiguous structures such as reality-fiction. The phenomenon of the transitional area, discussed by Donald W. Winnicott, is the cultural manoeuvring space in which the subject experiences both internal and external reality while constructing the initial fiction of the self, and is closely related to Jean Baudrillard’s concept of simulation in which fiction and reality are mixed. In contemporary art, where the artistic image becomes a virtual reality, phantasmagoria, in which meaning and reality are reconstructed and staged in new media works referencing digital game practices, takes place in the transitional areas that Winnicott sees as a dynamic and experiential playground. This research focuses on digital game art works of different artists such as Olafur Eliasson, Hesam Jalili, Theo Triantafyllidis, Ian Cheng, and Joon Yong Moon, who create a transitional space in their work with practices such as ‘gamification of space’ and ‘counter gaming’.

AutoAMS: Automated attention-based multi-modal graph learning architecture search

Multi-modal attention mechanisms have been successfully used in multi-modal graph learning for various tasks. However, existing attention-based multi-modal graph learning (AMGL) architectures heavily rely on manual design, requiring huge effort and expert experience. Meanwhile, graph neural architecture search (GNAS) has made great progress toward automatically designing graph-based learning architectures. However, it is challenging to directly adopt existing GNAS methods to search for better AMGL architectures because of the search spaces that only focus on designing graph neural network architectures and the search objective that ignores multi-modal interactive information between modalities and long-term content dependencies within different modalities. To address these issues, we propose an automated attention-based multi-modal graph learning architecture search (AutoAMS) framework, which can automatically design the optimal AMGL architectures for different multi-modal tasks. Specifically, we design an effective attention-based multi-modal (AM) search space consisting of four sub-spaces, which can jointly support the automatic search of multi-modal attention representation and other components of multi-modal graph learning architecture. In addition, a novel search objective based on an unsupervised multi-modal reconstruction loss and task-specific loss is introduced to search and train AMGL architectures. The search objective can extract the global features and capture multi-modal interactions from multiple modalities. The experimental results on multi-modal tasks show strong evidence that AutoAMS is capable of designing high-performance AMGL architectures.

AdaFN-AG: Enhancing multimodal interaction with Adaptive Feature Normalization for multimodal sentiment analysis

In multimodal sentiment analysis, achieving effective fusion among text, acoustic, and visual modalities for enhanced sentiment prediction is a crucial research topic. Recent studies typically employ tensor-based or attention-based mechanisms for multimodal fusion. However, the former fails to achieve satisfactory prediction performance, and the latter complicates the computation of fusion between non-textual modalities. Therefore, this paper proposes the multimodal sentiment analysis model based on Adaptive Feature Normalization and Attention Gating mechanism (AdaFN-AG). Firstly, facing highly synchronized non-textual modalities, we design the Adaptive Feature Normalization (AdaFN) method, which focuses more on sentiment features interaction rather than timing features association. In AdaFN, acoustic and visual modality features achieve cross-modal interaction through normalization, inverse normalization, and mix-up operations, with weights utilized for adaptive strength regulation of the cross-modal interaction. Meanwhile, we design the Attention Gating mechanism that facilitates cross-modal interactions between textual and non-textual modalities through cross-attention and captures timing associations, while the gating module concurrently regulates the intensity of these interactions. Additionally, we employ self-attention to capture the intrinsic correlations within single-modal features. Subsequently, we conduct experiments on three benchmark datasets for multimodal sentiment analysis, with the results indicating that AdaFN-AG outperforms the baselines across the majority of evaluation metrics. Through research and experiments, we validate that AdaFN-AG not only enhances performance by adopting appropriate methods for different types of cross-modal interactions while conserving computational resources but also verifies the generalization capability of the AdaFN method.