Multimodal Analysis Approach Research Articles

Sounding of the idols. Visual deception and religious images in Brazil

This article is a visual analysis of “deceptive religious imagery” employed by the campaign teams of Bolsonaro and Lula during the 2022 Brazilian presidential race with the aim of garnering votes through networked disinformation and covert influence operations. Employing a multimodal analytical approach, the research delves into the intricate interplay between politics and religion, particularly through the lens of online engagement with deceptive imagery. It tracks the dissemination of deceptive religious imagery across digital and physical platforms, melding visual analysis with rigorous fact-checking methodologies. This investigation is based on online and off-line ethnographic visual research. Not only this research let us reassess key concepts like ‘false image’, ‘verification’ or ‘visual myth’ but also allowed us to identify and debunk false imagery propagated online. Drawing upon Mitchell's framework of ‘iconology’, the article advocates for a critical examination of images, underscoring their tangible influence and the imperative to expose and evaluate their falseness in shaping public opinion. The research stresses the significance of understanding the actions and repercussions of religious visual disinformation in the context of the prevalence of post-truth cultures. In sum, this article contributes to our understanding of the intricate dynamics surrounding the utilisation of deceptive religious imagery in political campaigns, shedding light on the broader implications for democratic processes and the manipulation of public perception in contemporary socio-political contexts.

Multimodal analysis of methylation and fragmentomic profiles in plasma cell free DNA for differentiation of benign and malignant breast tumors.

7 Background: Breast cancer ranks as the second leading cause of cancer-related mortality among women globally. Early detection of breast cancer is crucial for improving patient outcomes and reducing mortality rates. Liquid biopsy, which relies on circulating tumor DNA (ctDNA) shed by breast tumors into the bloodstream, presents a promising non-invasive approach for early breast cancer detection. However, accurately distinguishing between benign breast abnormalities and malignant tumors remains a significant clinical challenge, as misdiagnosis can lead to unnecessary invasive procedures. Methods: Herein, we employed a multimodal analysis approach, namely SPOT-MAS (Screen for the Presence of Tumor by DNA Methylation and Size), to profile alterations in methylation and fragment length patterns of cell-free DNA (cfDNA) from 169 breast cancer-confirmed patients and 99 patients diagnosed with benign breast lumps including cysts, fibroadenomas, and fibrocystic changes. A robust machine learning model was constructed using these signatures to differentiate between breast cancer patients and individuals with benign lesions. Results: Our genome-wide analyses identified distinct profiles of methylation changes, copy number alterations, and end motifs in cfDNA, enabling discrimination between breast cancer patients and individuals with benign conditions. Notably, we observed a significant enrichment of Thymine and Adenine at cfDNA cleavage sites in breast cancer patients. Moreover, our target sequencing analyses uncovered distinct methylation patterns in the regulatory regions of multiple genes, including hypermethylation in GPR126 or hypomethylation in TOP1 or MAFB in breast cancer cfDNA. Our multi-featured model achieved an AUC of 0.92 (95% CI: 0.88–0.97), a specificity of 97.44% and sensitivities of 57.14% and 61.70% for stage I and stage II–III patients, respectively. Furthermore, our multimodal assay effectively differentiated multiple molecular subtypes of breast tumors from benign lesions, achieving the highest sensitivity of 71.43% with Luminal A, followed by Luminal B, Triple Negative Breast Cancer (TNBC), Luminal B-HER2, and HER2 with sensitivities of 66.67%, 60%, 58.33% and 55.56%, respectively. Conclusions: Our findings demonstrate the potential of cancer-specific methylation and fragmentomic patterns in plasma cfDNA as novel biomarkers for accurately discriminating between breast cancer and benign lesions. This capability reduces the false-positive rate and helps avoid unnecessary biopsies in current clinical practice.

Using multimodal learning analytics as a formative assessment tool: Exploring collaborative dynamics in mathematics teacher education

AbstractBackgroundTraditionally, understanding students' learning dynamics, collaboration, emotions, and their impact on performance has posed challenges in formative assessment. The complexity of monitoring and assessing these factors have often limited the depth and breadth of insights.ObjectivesThis study aims to explore the potential of multimodal learning analytics as a formative assessment tool in math education. The focus is on discerning how collaborative discourse behaviours and emotional indicators interplay with lesson evaluation performance.MethodsUsing undergraduate students' multimodal data, which includes collaboration data, facial behaviour data, and emotional data, the study explored the patterns of collaboration and emotion. Through the lens of multimodal learning analytics, we conducted exploratory data analysis to identify meaningful relationships between specific types of collaborative discourse, facial expressions, and performance indicators. Moreover, the study evaluated a machine learning model's potential to predict target learning outcomes by integrating data from multiple channels.ResultsThe analysis revealed key features from both discourse and emotion data as significant predictors. These findings underscore the potential of a multimodal analytical approach in understanding students' learning process and predicting outcomes.ConclusionsThe study emphasizes the importance and feasibility of a multimodal learning analytic approach in the context of math education. It highlights the academic and practical implications of such an approach, along with its limitations, pointing towards future research directions in this area.

A multimodal analytical approach is important in accurately assessing terpene composition in edible essential oils

Terpenes in essential oils (EOs) have recently received significant attention due to their potential to improve brain and whole-body health. A deeper understanding of the terpene composition of edible EOs is important for fully exploring their possible applications. In our study, we employed a comprehensive study using four different methods to analyze EO samples, including GC–MS with solid phase microextraction (SPME), liquid injection (LI), derivatization to trimethylsilyl ethers (TMSE), and LC-MS. Our findings revealed that relying on a single analytical method may be insufficient for detecting all terpenes in EOs. Despite identifying a total of 156 terpenes in the samples, only 58 were detectable across all 4 methods. To obtain a more accurate terpene profile of EOs, we advocate for the combined use of LI-GC and TMSE-GC. The terpenes detected by these two methods are complementary, enabling the detection of all terpenes with high VIP in the samples.

Predicting teachers’ sense of efficacy: A multimodal analysis integrating SEM, deep learning, and ANN

AbstractThis study aims to investigate the predictive role of cultural intelligence, motivation to teach, and “culturally responsive classroom management self‐efficacy” (CRCMSE) in teachers’ sense of efficacy. The study utilized a combination of “structural equation modeling” (SEM), deep learning, and “artificial neural network” (ANN) to analyze data collected from 1061 preservice teachers. The SEM analysis indicated that cultural intelligence, motivation to teach, and CRCMSE significantly predicted the sense of efficacy of the teacher candidates, accounting for 59% of the variance. Additionally, the ANN model accurately predicted the teachers’ sense of efficacy with 75.71% and 75.17% accuracy for training and testing, respectively. The sensitivity analysis revealed that CRCMSE played the most crucial role in predicting the preservice teachers’ sense of efficacy. The deep learning model also predicted the sense of efficacy with an overall accuracy of 74.18%. The utilization of a multimodal analysis approach facilitated the identification of both linear and nonlinear relationships between the constructs.

Tensions of place and opposing emblems of authenticity and marginal gentrification in Grünerløkka, Oslo

This study investigates multiple discourses surrounding the trends, changes and opposing emblematic signage of authenticity and the ideological constructions of place within Grünerløkka, Oslo, Norway. Our theoretical framework is grounded within the political economy of place and the sociolinguistics of authenticity. Three different datasets are used in the analysis: a) 60 interviews conducted with local residents, business owners and journalists; b) multilingual and multimodal signage found within this neighborhood's changing linguistic and semiotic landscapes; and c) texts from online media. We take both a discourse and multimodal analytic approach in our investigation of the circulating tensions of place found both ‘online’ and ‘offline’ of what constitutes an ‘authentic’ neighborhood to individuals. Our findings suggest that the conflicting ideologies of authenticity are bound to the pressures of change and resistance. These are inextricably connected to local consumer practices that are inevitably tied to global market forces and marketing strategies within the social and material constructions of urban landscapes.

A multimodal analytical approach is important in accurately assessing terpene composition in edible essential oils

A multimodal analytical approach is important in accurately assessing terpene composition in edible essential oils

A Study on the Value of Translated Texts in Business Japanese Literature Based on Multimodal Analysis Approach

Abstract This study focuses on the textual value of business Japanese literary translations. By employing multimodal analysis, this paper extracts distinctive text features. It incorporates feature layer fusion to seamlessly integrate features from various modalities using an attention mechanism. These integrated features are then processed by a multilayer perceptron, which aids in classifying text-based emotions and extracting relevant emotional data. A specific research case has been selected to explore the communicative and emotional impact of translated texts from the audience’s perspective. The results show that in different modalities, the mean value of the print path is the highest (4.4056), and the movie and television path is the lowest (4.0749) in terms of audience preference. The book’s content is the primary focus of the audience. In terms of emotional value, the audience not only thinks about the obvious thoughts about “life (64)”, “love (143)”, and “sacrifice (74)” in the story but also about “faith” under these thoughts. Faith”, “humanity,” “justice,” “salvation,” and “betrayal,” “betrayal”.

A Study of Mainstream Media Communication and Audience Participatory Culture Based on Multimodal Analysis Approach

Abstract This paper analyzes the network expression and audience participatory discourse system under the mainstream media communication mode, combines the audiovisual multimodal audience cognitive participation and the multimodal features of the mainstream media, and proposes a multimodal deep learning model based on mainstream media communication. Based on the picture vision and human emotion migration methods, the picture vision feature migration module and the text emotion migration module are constructed respectively, and the multimodal feature fusion model is built by applying the gradient enhancement decision tree. Linguistic modality analysis of mainstream media communication discourse is carried out by analyzing the development characteristics of audience participatory cultural communication modes. The influence of different mainstream media communication discourses on the audience’s participatory culture is examined through logistic regression analysis while screening network discourse access. The regression analysis shows that the mainstream media communication modes represented by social software and microblogging have a significant impact on the development of the audience’s participatory culture, with p-values of 0.0452 and 0.0068, respectively, and thus the mainstream media need to strengthen content innovation, focus on specific audience groups, and make full use of new media platforms to provide audiences with richer, more accurate and professional news services.

Exploring WhatsApp as Teaching and Learning Activities during Covid-19/New Normal era: A Semiotic Technology Analysis

In the landscape of modern education, the integration of technology, particularly within language learning, has gained substantial scholarly attention. Among digital platforms, WhatsApp has undergone a notable transformation, evolving from a basic messaging application to a versatile social hub. Its role expanded significantly during the Covid-19 pandemic, becoming a pivotal tool for online teaching and learning activities, prevalent not only in higher education but also across secondary levels. While existing studies have predominantly focused on user interaction within WhatsApp, limited attention has been directed toward the technical aspects or the application’s usage in the teaching and learning process. This research aims to comprehensively explore WhatsApp’s potential as a facilitator for English language studies, employing a semiotic multimodal analysis approach guided by Poulsen’s seminal work on semiotic technology. The study delves into WhatsApp’s intricate network of semiotic resources, semiotic regimes, and interactions, unveiling the complex interplay between language elements, multimodal features, and cultural aspects within this digital realm. Through the lens of semiotics, multimodal studies, and social semiotics, this investigation endeavors to offer a holistic understanding of WhatsApp’s use as an educational tool, shedding light on its multifaceted role in contemporary educational contexts.

Research on English Foreign Propaganda Translation and External Communication Paths Based on Multimodal Analysis Approach

Abstract In this paper, a multimodal phrase corpus is created, enabling phrases to be searched automatically. The phrase structure is constructed through phrase centroids, and multilevel linguistic measurements are obtained for vocabulary, syntax, and parts of speech styles. A joint feature analysis model for the automatic conversion of English-Chinese machine translation is constructed through distributed feature fusion and joint parameter analysis. Linear statistical features and feature analysis detection are utilized to compute the optimal solution for English-Chinese machine automatic translation text. The graph embedding technique was used to pretrain the knowledge graph and obtain information about the network structure. Combined with the graph convolution technique, deep semantic information mining was carried out to achieve efficient content matching and cultural dissemination. It has been proved that the method of this paper has an AUC value of 0.921 and a satisfaction rate between 80% and 95% in English-translated movie data. The proposed method can accurately translate information from English to foreign propaganda content, which further promotes the development of international cultural exchange and communication.

Gender Stereotypes in TV Commercials: A Multimodal Analysis Approach

The current study examines gender stereotypes in TV commercials through the lens of multimodality. It adopts Kress and Van Leeuwen's (2006) visual grammar to decipher the visual resources that form gendered depictions in Algerian TV advertising discourse. To achieve this aim, five TV ads broadcasted on public and private Algerian TV channels were selected and downloaded from YouTube, then they were analyzed in terms of the representational metafunction, the interactive metafunction, and the compositional metafunction. Results show that Algerian TV commercials are loaded with offensive depictions for both genders. They also indicate that different multimodal resources are involved in the construction of these gendered portrayals. The study concludes with some recommendations for ad designers and TV producers.

Application of 4-D ultrasound-derived regional strain and proteomics analysis in Nkx2-5-deficient male mice.

The comprehensive characterization of cardiac structure and function is critical to better understanding various murine models of cardiac disease. We demonstrate here a multimodal analysis approach using high-frequency four-dimensional ultrasound (4DUS) imaging and proteomics to explore the relationship between regional function and tissue composition in a murine model of metabolic cardiomyopathy (Nkx2-5183P/+). The presented 4DUS analysis outlines a novel approach to mapping both circumferential and longitudinal strain profiles through a standardized framework. We then demonstrate how this approach allows for spatiotemporal comparisons of cardiac function and improved localization of regional left ventricular dysfunction. Guided by observed trends in regional dysfunction, our targeted Ingenuity Pathway Analysis (IPA) results highlight metabolic dysregulation in the Nkx2-5183P/+ model, including altered mitochondrial function and energy metabolism (i.e., oxidative phosphorylation and fatty acid/lipid handling). Finally, we present a combined 4DUS-proteomics z-score-based analysis that highlights IPA canonical pathways showing strong linear relationships with 4DUS biomarkers of regional cardiac dysfunction. The presented multimodal analysis methods aim to help future studies more comprehensively assess regional structure-function relationships in other preclinical models of cardiomyopathy.NEW & NOTEWORTHY A multimodal approach using both four-dimensional ultrasound (4DUS) and regional proteomics can help enhance our investigations of murine cardiomyopathy models. We present unique 4DUS-derived strain maps that provide a framework for both cross-sectional and longitudinal analysis of spatiotemporal cardiac function. We further detail and demonstrate an innovative 4DUS-proteomics z-score-based linear regression method, aimed at characterizing relationships between regional cardiac dysfunction and underlying mechanisms of disease.

Embodied meaning‐making: using literacy‐as‐event to explore a young child's small world play

AbstractThis article uses the concept of literacy‐as‐event to explore the embodied meaning‐making of a young child during small world play. Recent developments in literacy research, influenced by relational thinking, have led to a reconsideration of how meaning‐making unfolds in home and school settings. The concept of literacy‐as‐event suggests that meaning‐making is unpredictable and dynamic, responding to novel socio‐material interactions between texts, people, objects and moments. This view suggests that there is a need to ensure children have opportunities to engage with embodied and material meaning‐making beyond shared reading events. In this article, small world play after a shared reading event is positioned as enabling socio‐material meaning‐making through embodied and material encounters with people and objects. A single episode of small world play is presented for analysis. A multimodal analytical approach is used, drawing attention to the embodied interactions between a child, her adult and objects. Analysis of the data shows that the young child's meaning‐making involved moments of physical and material almost‐hiatus, followed by erratic movements. These often unexpected fluctuations, between stillness and motion, created generative tensions between the child and her adult, enabling creative swerves in engagement between narrative action, character traits and story themes.

Psychophysiological Parameters Predict the Performance of Naive Subjects in Sport Shooting Training.

In this study, we investigated the neural and behavioral mechanisms associated with precision visual-motor control during the learning of sport shooting. We developed an experimental paradigm adapted for naïve individuals and a multisensory experimental paradigm. We showed that in the proposed experimental paradigms, subjects trained well and significantly increased their accuracy. We also identified several psycho-physiological parameters that were associated with shooting outcomes, including EEG biomarkers. In particular, we observed an increase in head-averaged delta and right temporal alpha EEG power before missing shots, as well as a negative correlation between theta-band energies in the frontal and central brain regions and shooting success. Our findings suggest that the multimodal analysis approach has the potential to be highly informative in studying the complex processes involved in visual-motor control learning and may be useful for optimizing training processes.

Functionalized Clays for Radionuclide Sequestration: A Review

Performance assessment of nuclear waste disposal options requires implementation of effective buffer materials. Buffer materials must meet longevity requirements and scavenge challenging radioisotopes, e.g., iodine-129 (I-129) and technetium-99 (Tc-99), both long-lived, highly mobile, anionic species in oxic environments. In many proposed nuclear waste repositories, heat generating radioactive waste will be surrounded by clay buffer material, which swells to fill the gap between the waste package and the host geology, restricting radionuclide transport to diffusive processes for several hundred years. The clay can be functionalized to further the limit release of Tc-99, as pertechnetate (TcO4–), and I-129, as iodate (IO3–) and iodide (I–). Here, we review clay buffer functionalization and the potential to enhance sequestration of anionic radionuclides that have an increased risk of leaching out of a disposal facility. Functionalization with quaternary amines, nanosilver, bismuth-based materials, silane, and aluminum pillars have been demonstrated to improve the removal of targeted contaminants from solution. This review defines key research questions that remain: (i) can functionalized clay immobilize TcO4–, IO3–, and I– to the same extent as the reactive components themselves? (ii) Does functionalization affect the ability of the clay barrier to perform its safety functions of protecting the waste canister and limiting radionuclide transport? In addition to radionuclide retention, a multimodal analytical approach is required to assess functionalized clay performance, and recommended techniques are reviewed here. The key parameters that can influence radionuclide sequestration by functionalized clay are also evaluated. Theoretical approaches, including both density functional theory calculations and molecular dynamics simulations, can inform the energetics and kinetics of radionuclide interactions with functionalized clays. This combined experimental and theoretical approach can link atomistic and microscopic processes to predict the macroscopic physical properties of the functionalized clays.

Abstract 1711: A multi-modal analysis approach leveraging multiplexed spatial phenotyping and multi-omics analysis to better understand the prognostic value of tertiary lymphoid structures in non-small cell lung cancer

Abstract Introduction: Tertiary Lymphoid Structures (TLS) are highly organized ectopic lymphoid structures found in inflamed or tumor tissues, acting as sites of lymphoid recruitment and immune activation. A high TLS density within the tumor is commonly associated with an increased prognostic effect of TILs and with an improved disease free survival and overall survival for patients. However, the existence of conflicting studies suggest that multiple TLS features should be taken into account when assessing their prognostic value, such as their location, cellular composition, maturation stage and spatial organisation, as those may affect their functionalities. Methods: With the aim of gaining insights into TLS biology and evaluating the prognostic role of TLS in Non-Small Cell Lung Carcinoma (NSCLC) according to their multiple features, we developed a TLS multiplex immunofluorescent (mIF) panel that includes T cell (CD3, CD8), B cell (CD20), Follicular Dendritic cell (CD21, CD23) and mature dendritic cell (DC-LAMP) markers. We deployed this panel across a cohort of primary tumors from NSCLC patients (n=408) and established a mIF image analysis workstream to assess the status and spatial location of each cell within the tissue. A H&E staining of the same tissue section was performed to evaluate mIF spatial data in relation to the tumor context. Additional multi-omics assessments were conducted across the same cohort including; whole exome sequencing, NanoString transcriptomics, and immunohistochemistry (e.g. PD-L1, FOXP3). We have leveraged clinical metadata, including demographics (e.g. age, sex, smoking status) and clinical risk factors (e.g. stage, grade, Standard of Care treatment) with clinical follow up (e.g. OS, PFS) for prevalence analysis, novel biomarker identification, and survival association. Results: Assessment of the prevalence of each cell phenotype within the tumor tissue and TLS (tumor centre vs invasive margin; tumor epithelium vs stroma), the distance between each cell type, and the distance of non-TLS immune cells to the closest TLS will be described, demonstrating the different types of lymphoid aggregates and TLS and their functional status. An integrative analysis combining these spatial biology data with multi-omics and clinical data will be presented evaluating the prognostic value of TLS composition, maturation status and spatial organization, in correlation with additional biomarkers and clinical characteristics. Conclusion: This exploratory study using cutting-edge technologies enables us to better understand how TLS orchestrate an organised anti-tumour response, defining TLS spatial biomarker signatures, TLS gene signatures, and TLS features associated with NSCLC patient outcomes to evaluate in the clinic. Citation Format: Julie Berthe, Felix Segerer, Emily C. Jennings, Alma Andoni, Marco Testori, Megha Saraiya, Miljenka Vuko, Harald Hessel, Andreas Spitzmüller, Mari Heininen-Brown, Jorge Blando, Felicia Ng, Emma Jones, Sophie Willis, Michael Surace, Rieneke van de Ven, Tanja De Gruijl, Helen Angell. A multi-modal analysis approach leveraging multiplexed spatial phenotyping and multi-omics analysis to better understand the prognostic value of tertiary lymphoid structures in non-small cell lung cancer [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2022; 2022 Apr 8-13. Philadelphia (PA): AACR; Cancer Res 2022;82(12_Suppl):Abstract nr 1711.

A Multimodal Discourse Analysis of Some Newspapers Campaign Advertisements for Nigeria’s 2019 Elections and Their Appeal Strategies

This study investigated the multimodal resources of some political campaign ads for the 2019 general elections in Nigeria with the purpose of analyzing their appeal strategies. The objectives were to: examine the several communicative elements of the 2019 elections campaign advertisements as pieces of discourse as well as apply the analytical tool of Discourse Analysis to establish the appeal strategies adopted to achieve some political objectives. Data for this study were drawn from newspaper advertisements in three national dailies: the Punch, the DailySun and the Vanguard. The newspapers were published between December 2018 and February 2019. Six advertisements were purposively selected for this study owing to the several rhetorical appeals in them. The analysis of data was based on discourse multimodal analysis approach. The analysis showed that the multimodal resources of the ads, which were socio-culturally grounded, were analyzed under the broad themes of appeal to performance, appeal to endorsement and appeal to freedom which were the major tropes in the ads. In conclusion, it was implied in the study that the multimodality study is an important part of political expression generally and in Nigeria specifically, mere looking at the fact that the semiotic resources deployed in political campaign ads on newspapers convey uncommon yet compelling and significant political meanings and messages about politicians and so on.

Affective governance as multimodal discursive practice in Singapore’ COVID-19 vaccination video

Drawing on scholarship on affective-discursive practice and employing a critical multimodal analytic approach, this paper examines how an official vaccination campaign video from Singapore affectively engages with audiences through the construction of various social relationships and identities. Analysis of the multimodal images identifies four salient themes in the video: (1) a harmonious neighborhood comprising inter-ethnic friendships, (2) a paternalistic father-child relationship, (3) a male-dominant family model, and (4) a nostalgic past-present relationship. The analysis shows that affect is discursively mobilized in the orchestration of embodied modes, auditory, and visual modes via strategic filming and editing techniques. Based on the analysis, the authors argue that the video not only functions to subtly persuade the public to get vaccinated but should also be seen as part of a broader move toward an ‘affective politics’ which aims to mobilize the public, reinforce social cohesion and manage the global pandemic crisis in contemporary Singapore.

Positioning of female marriage immigrants in South Korea: a multimodal textbook analysis

Abstract This study examines a series of government textbooks used to teach Korean language and culture to spouses of South Korean nationals living in Korea. A multimodal analysis of textbook images and narratives explores how strictly defined gender identity discourses are constructed and circulated through these government textbooks in ways that not only seek to reinforce gender conventions, but also advance a conservative, assimilationist curriculum for these immigrants. The analysis reveals ways in which female marriage immigrants are positioned primarily as housekeepers; how they are subordinately positioned vis-à-vis in-laws and their husbands; and finally, how intercultural miscommunications are presented as the fault only of the immigrants. In this, the burden for intercultural communication and learning is placed solely on the immigrants in the multimodal narratives presented in the textbooks, without reciprocal burden ever placed on the spouse or spouse’s family, or effective ways of negotiating intercultural miscommunications being shared. Images and text work together in multimodal ensembles to craft narratives that position immigrant women in ways that textual or visual analysis alone would not fully capture. Thus, the study argues the use of multimodal analytical approach in examining language textbooks as a way of deconstructing broader discourses and disrupting problematic discourses.