Multimodal Analysis Research Articles

NLP-Based Extended Lexicon Model For Sarcasm Detection With Tweets And Emojis

Sarcasm detection in social media, especially on platforms like Twitter, poses a significant challenge due to the informal and context-dependent nature of language. This research presents an NLP-based extended lexicon model for sarcasm detection, leveraging both textual features and emojis to enhance interpretability and accuracy. The proposed model integrates a sentiment lexicon with syntactic and semantic cues, enriched by emoji sentiment analysis, to capture subtle contradictions and ironic tones in tweets. A dataset of labeled tweets is preprocessed and analyzed using natural language processing techniques, including tokenization, POS tagging, and feature extraction. Machine learning classifiers are then employed to detect sarcasm. Experimental results demonstrate that the inclusion of emoji sentiment and lexicon-based features significantly improves detection performance, highlighting the importance of multimodal analysis in understanding sarcasm in social media texts.

PRE-SERVICE TEACHERS’ CONCEPTIONS OF “BEING GIFTED” AND “BEING TALENTED”: A MULTIMODAL SEMIOTIC APPROACH

Teachers’ conceptions of gifted and talented students significantly influence their nomination skills and teaching practices. However, research indicates that these conceptions are often incorrect or inconsistent, even among teachers who have completed coursework in gifted and talented education. This study aimed to explore pre-service teachers’ conceptions of “being gifted” and “being talented” using a qualitative case study research design with 35 participants. Data were collected using a multimodal instrument that allowed participants to express their mental representations through various modes such as writing, drawing, and mathematical symbols. The collected data were then analyzed using a multimodal semiotic analysis protocol developed specifically for this purpose. The results revealed that participants predominantly associated “being gifted” with cognitive abilities such as high IQ, scientific inquiry, and mathematics, while “being talented” was linked to cognitive, affective, and psychomotor abilities, including high performance in arts and sports. The findings highlight the need for pre-service training programs that promote scientific conceptions of giftedness and talent, addressing stereotypes and lay misconceptions. Keywords: gifted education, gifted students, multimodal analysis, talented students, teacher conception

MRI-based brain structural changes in adult-onset dentatorubral-pallidoluysian atrophy.

Adult-onset dentatorubral-pallidoluysian atrophy (DRPLA) is a rare autosomal dominant spinocerebellar ataxia subtype, characterized by widespread brain neurodegeneration. However, comprehensive studies on whole-brain structural changes remain limited. This study aimed to quantify brain structural damage in adult-onset DRPLA and correlate these findings with clinical data. Twenty-four adult-onset DRPLA patients were recruited and underwent structural and diffusion MRI at 3.0 Tesla. Multimodal analyses were conducted to assess cortical thickness, the volumes of subcortical structures, the brainstem, cerebellum, and cerebral white matter (WM), as well as the microstructural integrity of WM. Results were compared with those from 30 age- and gender-matched healthy controls. The relationships between neuroimaging abnormalities and clinical data, including ataxia severity, cognitive performance, CAG repeat length, and disease duration, were explored. In adult-onset DRPLA patients, reduced cortical thickness was observed in the bilateral frontal, parietal, temporal, and occipital lobes. Volumetric analysis revealed significant reductions in all subcortical structures, as well as in the brainstem, cerebellum, and cerebral WM substructures. Tract-based spatial statistics (TBSS) analysis of diffusion tensor imaging (DTI) and neurite orientation dispersion and density imaging (NODDI) metrics demonstrated widespread WM abnormalities throughout the brain. Furthermore, several imaging measures demonstrated significant correlations with ataxia severity, cognitive function scores, CAG repeat length, and disease duration. DRPLA should be considered a multisystem neurodegenerative disease, characterized by widespread and severe gray matter and WM atrophy throughout the brain. The advanced MRI techniques employed in this study expand our understanding of the underlying pathophysiological mechanisms.

Reconstruction of the Chronology of Kings in the Early Egyptian Dynasty: A Multimodal Analysis

This comprehensive study delves into the historical enigma of Egyptian Predynastic Kings, employing a meticulous multi - approach analysis. Utilizing archaeological evidence from the Palermo Stone and the Turin Papyrus, the methodology establishes a novel framework for dating these ancient rulers. By amalgamating historical records, mathematical calculations, and climatic context, the paper elucidates the inaugurational timelines of key Predynastic monarchs. The findings not only shed light on the Predynastic Kings era but also foster insights into its alignment with mythological epochs and climatic transitions. The study underscores the significance of diverse validation methods and culminates in a comprehensive chronology of these enigmatic rulers.

A Multi-modal Analysis of the Impact of Subtitle Translation in the American Version of "Empresses in the Palace" on Cultural Dissemination

In the context of globalization, the rise of Chinese historical dramas on the world stage offers rich material for studying the multi-modal role of subtitle translation, exemplified by the American version of "Empresses in the Palace." This paper delves into the impact of subtitle translation strategies on cultural dissemination within this drama using multi-modal theory. The drama, set against the backdrop of the imperial harem of the Qing Dynasty, chronicles the growth of Zhen Huan. Following its success in Asia, adjustments were made to the American version to align with the viewing preferences of American audiences. Through multi-modal analysis, it is revealed that within the visual modality, costumes and settings facilitate translation comprehension. In the auditory modality, synchronization challenges need addressing, while the voice characteristics of characters influence subtitle choices. Furthermore, translators must prioritize cultural differences when considering the interplay between textual and cultural modalities. Subtitle translation exerts a positive influence on cultural dissemination, facilitating the promotion of traditional Chinese culture, the spread of the Mandarin language, and the fostering of cultural self-confidence. Nevertheless, this study acknowledges its limitations. Future research endeavors can further advance the globalization of Chinese culture by deepening the exploration of cultural elements, broadening the sample scope, and incorporating quantitative analysis.

Multi-task learning framework using tri-encoder with caption prompt for multimodal aspect-based sentiment analysis

Multi-task learning framework using tri-encoder with caption prompt for multimodal aspect-based sentiment analysis

Multi-grained feature gating fusion network for multimodal sentiment analysis

Multi-grained feature gating fusion network for multimodal sentiment analysis

Tebentafusp elicits on-target cutaneous immune responses driven by cytotoxic T-cells in uveal melanoma patients.

Tebentafusp is the first T-cell receptor-based bispecific protein approved for clinical use in HLA-A*02:01+ adult patients with unresectable/metastatic uveal melanoma. It redirects T-cells toward gp100-expressing target cells, frequently inducing skin-related early adverse events. This study investigated immunological and cellular responses using single-cell and spatial analysis of skin biopsies from patients with metastatic uveal melanoma treated with tebentafusp. 81.8% of patients developed acute cutaneous adverse events, which correlated with improved survival. Multimodal analysis revealed a brisk infiltration of CD4+ and CD8+ T-cells, while melanocyte numbers declined. Single-cell RNA-sequencing revealed T-cell activation, proliferation, and IFN-γ/cytotoxic gene upregulation. CD8+ T-cells co-localized with melanocytes and upregulated LAG3, suggesting potential for combination therapies with tebentafusp. Melanocytes upregulated antigen presentation and apoptotic pathways, while pigmentation gene expression decreased. However, gp100 remained stably expressed. Sequential skin biopsies enable in vivo pharmacodynamic modeling of tebentafusp, offering insights into immune activation, toxicity, and treatment response. Examining the on-target effects of bispecifics in tissues amenable to longitudinal sampling enhances our understanding of toxicity and therapeutic escape mechanisms, guiding strategies for treatment optimization.

The Evolution of Cartoon Images of Pandas in Western Media: A Multimodal Analysis

The Evolution of Cartoon Images of Pandas in Western Media: A Multimodal Analysis

On stance-taking with one-sided vs. two-sided shoulder lifts in German talk-in-interaction.

Taking a stance toward events, objects, and other persons is fundamental to human interaction. We investigate one specific body movement that is involved in stance-taking in interaction: a shoulder lift, realized as either a one-sided or a two-sided movement. Using multimodal Conversation Analysis, we trace how interactants employ shoulder lifts in different positions within responsive turns in various interaction types in German. This study reveals how the actions to which shoulder lifts contribute are bound to specific turn and sequence positions. We demonstrate how shoulder lifts are used for disclaiming the speaker's accountability or responsibility by framing their turn as non-expandable or non-expansion-worthy, thus curtailing the sequence. Furthermore, the study shows how participants orient to different types of shoulder movements, i.e., lifts with one or with both shoulders, as accomplishing different interactional tasks. By showing that shoulder lifts are a positionally sensitive resource for speakers in building stances, we showcase the potential of conversation analytic and interactional linguistic approaches to further our understanding of multimodal stance-taking in interaction.

Epigallocatechin Gallate Induces miR-192/215 Suppression of EGR1 in Gastric Cancer.

Gastric cancer (GC) is a leading cause of cancer-related deaths worldwide. MicroRNAs (miRNAs) are key regulators of tumorigenesis. This study investigated the role of the miR-192/215-early growth response protein 1 (EGR1) axis in GC and explored its therapeutic implications. Combinatorial molecular approaches including dual-luciferase reporter assays and western blot analyses were employed to authenticate EGR1 as a direct target of miR-192/215. Multi-modal data analysis incorporating The Cancer Genome Atlas datasets and our institutional cohort (n=56) was conducted to delineate EGR1 expression profiles across clinicopathological GC stages. Pharmacological modulation studies were implemented to characterize the dose- and time-dependent effects of epigallocatechin-3-gallate (EGCG) on the miR-192/215-EGR1 axis. EGR1 is a direct target of miR-192/215, with its expression negatively regulated by these miRNAs. Clinical analysis revealed significantly reduced EGR1 expression in tumors of early-stage GC (stage I) and cases with superficial invasion depth (T1) (p<0.05), suggesting a tumor-promoting role in the early stages of the disease. Higher EGR1 levels were associated with poorer overall survival, highlighting its potential as a prognostic biomarker (p<0.05). Furthermore, treatment of cells with EGCG transiently up-regulated miR-192/215 and down-regulated EGR1, highlighting its therapeutic potential. This study highlights the miR-192/215-EGR1 axis as a critical regulator of GC progression and a promising therapeutic target. EGCG may serve as an adjunct therapy for GC. Future studies should focus on the regulatory mechanisms of EGR1, its interaction with the tumor microenvironment, and clinical validation of EGCG and other agents targeting this axis.

Social media influencers’ transgressions – A multimodal critical discourse analysis of canceling strategies in threaded discussion on body shaming

Transgressions triggered by social media influencers’ inappropriate statements lead to social media hypes. The online public arena in Romania was dominated by a ‘stretch mark’ scandal when a famous influencer (George Buhnici) body shamed women. His controversial remarks urged a wave of outrage in the digitalized environment. Applying a sociocultural approach to character assassination and a Multimodal Critical Discourse Analysis, we aimed at identifying how the social media influencer (SMI) was represented in the online users’ comments and how the commenters discursively represented themselves. The main findings showed that name calling, ridicule, social identity denial, and disgracing were the four main character attack strategies present in the cancelation supporters’ representation of the influencer as the ‘other’. In their attempt to protect the body shamer (the influencer) from losing his social capital, cancelation resisters employed minimizing, bolstering, transcendence (freedom of speech), and attacking the accuser (the canceler) as main strategies to represent the influencer as ‘one of us’. Online users adopted three main roles in their discursive self-representation: delegitimators of collective Us, experiencers, and social advisers. In the end, the study provides a threefold relationship framework for the cancelation process triggered by social media influencers’ transgressions.

Semantic enhancement and cross-modal interaction fusion for sentiment analysis in social media.

The rapid development of social media has significantly impacted sentiment analysis, essential for understanding public opinion and predicting social trends. However, modality fusion in sentiment analysis can introduce a lot of noise because of the differences in semantic representations among various modalities, ultimately impacting the accuracy of classification results. Thus, this paper presents a Semantic Enhancement and Cross-Modal Interaction Fusion (SECIF) model for sentiment analysis to address these issues. Firstly, BERT and ResNet extract feature representations from text and images. Secondly, the GMHA mechanism is proposed to aggregate important semantic information and mitigate the influence of noise. Then, an ICN module is created to capture complex contextual dependencies and enhance the capability of text feature representations. Finally, a cross-modal interaction fusion module is implemented. Text features are considered primary, and image features are auxiliary, enabling the profound integration of textual and visual features. Themodel'sperformance is optimized by combining cross-entropy and KL divergence losses. The experiments are conducted using a dataset collected from public opinion events on Sina Weibo. The results demonstrate that the proposed model outperforms comparison models. The SECIF model improves by 11.19%, 82.27%, and 4.83% over the average accuracy of the text-only, image-only, and multimodal models, respectively. The proposed SECIF model is compared with ten baseline models on the publicly available datasets. The experimental results show that the SECIF model improves accuracy by 4.70% and F1 score by 6.56%. Through multimodal sentiment analysis, governments can better understand public emotions and opinion trends, facilitating more targeted and effective management strategies.

Translation In Early Turkish-German Illustrated Books For The Children Of Guest Workers: The Case Of Jugend&amp;Volk Verlag

This study lies at the intersection of narrative theory, multimodality, and materiality studies. It is argued that the intricate relations among linguistic, non-verbal, and multimodal elements acting as core modes and sub-modes in illustrated books contribute to a better understanding of the meaning-making process shaped by various probable conceptions of translation in illustrated books for children. This study focuses on the series of illustrated books for the children of guest workers published by Jugend&amp;Volk Verlag between 1972 and 1974 in Federal Germany. The corpus-based inquiry aims to explore the composition, operation, and correlations of the various modes of illustrated books for children. It is further discussed how translation conceptions based on the distinction of mode and medium serve to understand transfer mechanisms operating in the illustrated books for children. In doing so, it is proposed to integrate narrative and material elements in the multimodal analysis, thereby providing a multifaceted perspective on illustrated books. It is found out that illustrated books could be conceptualized as intramodal, intermodal, and intramedial translations with several modalities contradicting and/or complementing one another on various levels, which contributes to grasping distinct meaning-making processes emerging out of the complex interactions in the illustrated books.

Mobile Game Evaluation Method Based on Data Mining of Affective Time Series.

Our work is positioned at the intersection of game data science, affective gaming, and the implementation of multimodal body sensors analysis. We propose an original method of evaluating the quality of a class of video games based on the emotional reactions of players. Game developers ask why some games are more profitable (MP games) than others (LP games). An intuitively convincing hypothesis is often put forward: MP games evoke more positive emotions and hence are sustainably engaging. Our main hypothesis is that test players who can clearly distinguish between MP game and LP game in relatively short test sessions are more reliable in scoring games and valuable to keep track of their emotions. From a random group of test players, we selected players with such abilities. We analyzed their affective spectra and obtained a fairly clear confirmation that the selected players showed more positive and less negative emotions in MP games than in LP ones. We can reasonably expect these players to be focused on playing in the test session, and their emotions may really indicate the strengths of MP games over LP games. We present the results of the experimental evaluation of our method conducted with with a leading game company in Poland.

Alterations in Gray Matter Structure Linked to Frequency-Specific Cortico-Subcortical Connectivity in Schizophrenia via Multimodal Data Fusion.

Schizophrenia (SZ) is a complex psychiatric disorder that is currently defined by symptomatic and behavioral, rather than biological, criteria. Neuroimaging is an appealing avenue for SZ biomarker development, as several neuroimaging-based studies have shown measurable group differences in brain structure, as well as functional brain alterations in both static and dynamic functional network connectivity (sFNC and dFNC, respectively), between SZ and controls. The recently proposed filter-banked connectivity (FBC) method extends the standard dFNC sliding-window approach to estimate FNC within an arbitrary number of distinct frequency bands. Initial FBC results found that individuals with SZ spend more time in a less structured, more disconnected low-frequency (i.e., static) FNC state than HC, as well as preferential SZ occupancy in high-frequency connectivity states, suggesting a frequency-specific component underpinning the functional dysconnectivity observed in SZ. Building on these findings, we sought to link such frequency-specific patterns of FNC to covarying data-driven structural brain networks in the context of SZ. Specifically, we employ a multi-set canonical correlation analysis + joint independent components analysis (mCCA + jICA) data fusion framework to study the connection between gray matter volume (GMV) maps and FBC states across the full connectivity frequency spectrum. Our multimodal analysis identified two joint sources that captured co-varying patterns of frequency-specific functional connectivity and alterations in GMV with significant group differences in loading parameters between the SZ group and HC. The first joint source linked frequency-modulated connections between the subcortical and sensorimotor networks and GMV alterations in the frontal and temporal lobes, while the second joint source identified a relationship between low-frequency cerebellar-sensorimotor connectivity and structural changes in both the cerebellum and motor cortex. Together, these results show a strong connection between cortico-subcortical functional connectivity at both high and low frequencies and alterations in cortical GMV that may be relevant to the pathogenesis and pathophysiology of SZ.

Interpreting the Meaning of Lyrics and Music Video of "Sorai" by Nadin Amizah Based on Ferdinand de Saussure’s Theory

This study examines the interplay of lyrical and visual symbolism in Nadin Amizah’s Sorai through the lens of Ferdinand de Saussure’s semiotic theory. By analyzing the song’s lyrics and music video as multimodal texts, the research explores how linguistic and visual signifiers construct themes of interdependence, emotional burden, and acceptance of impermanence. Utilizing Saussure’s dyadic model of signifier-signified relationships, the analysis reveals that natural metaphors (e.g., sky, sea) and cultural lexicon (e.g., sorai, derived from Minangkabau) function as arbitrated codes to convey existential narratives. The lyrics emphasize emotional restraint and transience, while the music video amplifies these themes through chromatic symbolism, spatial binaries, and gestures of unattainable connection. Despite alignment in thematic exploration, the video diverges by universalizing impermanence through abstract visuals like tombstones and desaturated landscapes. The findings highlight the necessity of contextualizing Saussurean semiotics within cultural frameworks, particularly in non-Western artistic expressions, and demonstrate the dynamic role of multimodal analysis in deciphering layered meanings. This study contributes to broader discussions on the applicability of classical semiotics in contemporary media and underscores the interplay between regional idioms and universal human experiences.

Evolving Techniques in Fake News Detection: From Human Expertise to Large Language Models

The proliferation of fake news poses significant challenges in today’s digital age. This paper provides a comprehensive review of the most recent advancements in fake news detection methodologies, tracing their evolution from the earliest manual approaches to the latest state- of-the-art models, including large language models. The study identifies four key perspectives: knowledge based, style based, propagation based, and source based. Research on detection methods is classified into manual approaches and automatic approaches utilizing data science techniques like traditional machine learning, deep learning, and large language models. The review highlights the dual role of LLMs in generating and detecting fake news and discusses limitations of current methods, as the lack of datasets, emphasizing the need for multimodal analysis, interdisciplinary collaboration, and improved model transparency.

Key determinants of impaired gait performance in adults with neuromuscular diseases: a multiparametric and multimodal analysis.

Background. Neuromuscular diseases (NMD) are associated with gait impairments affecting quality of life. Contributing factors such as the severity of muscle weakness or kinematics alterations have mostly been studied separately. Therefore, the key determinants of impaired gait in patients with NMD remain unclear. This work aimed at investigating the major determinants of gait impairments through the assessment of muscle weakness, lower limb kinematic, muscle activation, and spatio-temporal parameters within a single session in patients and controls. Methods. Patients with NMD and controls were studied. After assessing maximal voluntary strength, subjects underwent a 2-minute walk test (2MWT) while recording electromyography, kinematics, and perceived exertion, dyspnea, and pain using verbal scales. A penalized regression was used to identify the main contributing variable to impaired gait performance and to build a predictive model for validation. Results. 28 with NMD and 17 controls were included. Eight major features of 2MWT gait performance were identified by the model: toe-off, stride length, step duration, cadence, ankle angle at foot strike, knee extensors strength, and timing variability in hip flexion/extension. The LASSO model provided accurate estimation of gait performance with 5.4% of average absolute error and 0.89 of R-squared. Substantial alterations in muscle activation patterns and lower limb kinematics were found in patients compared to controls. Conclusion. This study provides insights regarding gait impairments in patients with NMD. Identified determinants can guide targeted interventions to improve gait and quality of life. More research with larger cohorts and diverse conditions is needed to confirm and broaden these findings.

Framing and Reframing the Participation of the Communities, Groups, and Individuals Concerned: A Media Archaeological Analysis of Nomination Videos of the 2003 Convention Submitted by China

ABSTRACT This article examines how the participation of communities, groups, and individuals concerned (CGIs) in safeguarding intangible cultural heritage (ICH) is framed and reconfigured in 44 nomination videos submitted by China to UNESCO for inscription between 2008 and 2024. Employing media archaeology and multimodal analysis, it explores these videos as “archaeological” traces of heritage-making and layered dynamic “social text” shaped by institutional, political, and technological forces. While the analysis reveals a shift from state-centered narratives to increased CGIs visibility, it also demonstrates that this shift remains mediated through state-controlled framing, limiting grassroots agency. The study highlights how nomination videos function as heritage documentation and narrative construction tools, reinforcing risks of homogenization in global safeguarding practices. It concludes by advocating for CGIs-initiated narratives that prioritize empowerment and capacity building, proposing a more inclusive and sustainable approach to ICH representation.